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Abstract 


Individualized  instruction  significantly  improves  students  pedagogical  and  motivational 
outcomes.  In  this  paper,  we  seek  to  characterize  tutorial  behaviors  that  could  lead  to  these 
benefits  and  consider  why  these  actions  should  be  pedagogically  useful.  This  experiment 
examined  students  learning  LISP  programming  with  the  assistance  of  a  tutor.  Tutoring 
sessions  were  audiotaped.  allowing  us  to  analyze  every  verbal  utterance  during  the  ses¬ 
sions  and  thereby  identify  the  conversational  events  that  lead  to  pedagogical  success.  This 
discourse  analysis  suggests  that  tutors  are  successful  because  they  take  a  \ery  active  role 
in  leading  the  problem  solving  by  offering  confirmatory  feedback  and  additional  guidance 
while  students  are  on  profitable  paths  and  error  feedback  after  mistakes.  However,  tutors 
carefully  structure  their  feedback  to  allow  students  to  perform  as  much  of  the  work  as  pos¬ 
sible  while  the  tutor  ensures  that  problem  solving  stays  on  track.  These  results  suggest  the 
types  of  strategies  tutors  employ  to  facilitate  guided  learning  by  doing. 
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Introduction 

Novices  often  have  great  difficulty  mastering  new  domains.  It  is  generally  accepted  that 
the  best  wav  to  acquire  new  domain  skills  is  by  solving  problems  (Anderson.  1983.  Laird. 
Rosenbloom.  k  Newell.  1986;  VanLehn.  1988).  However,  there  are  dangers  inherent  in 
this  sort  of  learning  by  doing.  Floundering  during  problem  solving  often  leads  to  working 
memorv  overload,  which  interferes  with  learning  (Sweiler.  1988).  Furthermore,  errors  during 
problem  solving  can  often  lead  to  confusion  and  frustration.  It  is  very  difficult  to  learn  from 
problem  solving  episodes  that  consist  largely  of  attempts  to  recover  from  errors  (Anderson. 
1983:  Lewis  k  Anderson.  1985).  Ameliorating  these  costs  would  allow  students  to  gain  the 
maximum  benefits  of  learning  by  doing. 

Individualized  instruction,  or  tutoring,  considered  by  many  to  be  the  best  method  of 
instruction,  is  one  method  for  minimizing  .these  costs  (Bloom.  1984:  Cohen.  Kulik.  k  Ku- 
lik,  1982:  Lepper.  Aspinwall.  Mumme.  k  Chabay,  1990).  Individualized  instruction  has 
both  motivational  and  cognitive  benefits.  For  example,  tutoring  leads  students  to  feel  more 
competent  (Lepper  k  Chabay.  1988).  This  feeling  appears  to  be  justified:  tutored  students 
perform  two  standard  deviations  higher  than  their  colleagues  receiving  traditional  instruc¬ 
tion.  indicating  that  almost  all  tutored  students  perform  better  than  the  mean  performance 
of  students  receiving  classroom  instruction  (Bloom.  1984).  Yet  although  these  pedagogical 
benefits  have  been  noted,  their  source  has  not  been  adequately  characterized.  The  goal 
of  the  present  research  is  to  investigate  the  strategies  tutors  use  that  can  lead  to  these 
benefits. 

Merrill.  Reiser.  Ranney.  and  Trafton  (1992)  argued  that  tutors  balance  the  goal  of 
allowing  students  to  perform  as  much  of  the  problem  solving  as  possible  with  the  goal 
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of  ensuring  that  problem  solving  remains  productive.  Rather  than  letting  students  solve 
problems  on  their  own.  with  occasional  advice,  tutors  carefully  monitor  the  problem  solving 
to  ensure  that  it  stays  on  track  and  to  help  direct  students  back  towards  a  productive 
solution  path  when  needed. v  Thus,  tutors  offer  a  kind  of  guided  learning  by  doing  that 
allows  their  students  to  attain  the  benefits  of  learning  by  doing  while  avoiding  some  of 
the  costs.  In  this  paper,  we  examine  how  tutors  achieve  these  benefits  through  careful 
guidance  and  characterize  the  way  tutorial  interactions  lead  to  strong  learning  advantages 
for  students. 

We  will  describe  the  ways  in  which  tutors  guide  and  assist  student  learning  while  the\ 
solve  problems  and  work  to  understand  new  material.  Furthermore,  we  characterize  the 
situations  in  which  this  guidance  takes  place.  In  fact,  this  guidance  is  not  always  easy 
to  identify.  Examining  an  actual  tutoring  session  reveals  the  complexity  and  subtlety  of 
the  tutor's  role.  Table  1.  a  transcript  of  student-tutor  discussion,  shows  a  very  interactive 
relationship  between  student  and  tutor,  with  the  student  and  tutor  interrupting  each  other 
frequently  and  occasionally  completing  the  other  s  sentences. 

Insert  Table  1  about  here 

We  examine  tutorial  guidance  in  a  number  of  key  sites  for  learning.  First,  we  shall  see 
how  tutors  respond  when  students  are  on  an  appropriate  solution  path.  Perhaps  tutors 
concentrate  their  assistance  on  helping  students  realize  what  they  nave  done  correct!}  and 
understand  its  consequences.  For  example,  the  tutor  commented  that  a  step  was  correct  in 
lines  11.  17.  and  19  of  Table  1.  thus  encouraging  the  student  to  continue  with  that  path  in 
the  problem  solving,  and  elaborated  on  a  correct  student  action  in  line  9. 

Then  we  turn  to  an  examination  of  student-tutor  interactions  that  follow  a  problem 
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solving  difficulty.  Do  tutors  provide  feedback  after  impasses?  If  so.  what  form  does  the 
tutorial  guidance  take  —  are  tutors  very  directive  or  subtle?  Do  they  allow  students  to 
find  and  repair  their  own  mistakes,  offering  error  feedback  only  when  the  student  asks,  or 
instead  intervene  frequently  to  point  out  errors?  For  example,  in  line  number  4  of  Table  1. 
the  tutor  noticed  that  the  student  had  failed  to  include  ,  a  required  quotation  mark  and 
simply  told  the  student  how  to  fix  it  —  thus  not  allowing  the  student  to  find  the  error 
herself. 

The  goal  of  this  investigation  is  not  only  to  characterize  of  the  range  of  tutorial  strate¬ 
gies,  but  also  to  identify  the  factors  that  influence  when  tutors  intervene  and  the  intervention 
strategies  they  use.  It  should  be  possible  to  determine  these  factors  by  identifying  patterns 
and  consequences  of  tutorial  intervention.  Several  researchers  have  identified  tutorial  guid¬ 
ance  methods  (e.g.,  Fox.  1991;  Graesser.  Person,  Huber.  1993;  Lepper  et  ah.  1990;  Lepper 
Sz  Chabay,  1988;  Littman.  Pinto.  Solowav,  1990;  McArthur,  Stasz.  Sz  Zmuidzinas,  1990). 
By  and  large,  these  researchers  have  focused  on  a  few  central  episodes  that  occurred  during 
longer  tutoring  sessions.  Each  of  these  analyses  has  highlighted  some  of  the  characteristics 
of  tutorial  behavior,  thus  revealing  particular  aspects  of  tutorial  strategies  rather  than  a 
broad  range  of  behaviors  and  the  situations  in  which  they  arise.  Our  work  examines  behav¬ 
iors  in  a  larger  context  over  a  longer  period  of  time  to  capture  the  interplay  of  these  and 
other  tutorial  behaviors  with  student  problem  solving.  We  first  review  the  various  views 
of  tutoring  suggested  by  previous  research,  displaying  the  complexity  of  factors  affecting 
tutorial  behavior,  and  then  formulate  the  questions  that  drive  our  approach. 

Fox  (1991)  argued  that  tutors  provide  a  “safety  net '  for  students,  keeping  them  from 
going  off  track  by  offering  frequent  confirmatory  feedback.  Tutors  provided  a  confirmation 
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(e.g.,  "Yes")  to  each  student  step,  but  if  it  was  delayed  by  as  little  as  a  second  after  the 
step,  the  student  presumed  an  error  had  occurred  and  began  a  repair.  The  tutor  helped 
with  the  repair  as  needed,  even  to  the  extent  of  providing  the  correct  answer  if  necessary. 
However,  generally  tutors  tjped  to  remain  as  subtle  and  unobtrusive  as  possible.  Thus. 
Fox  characterized  feedback  as  primarily  though  not  completely  confirmatory,  keeping  the 
student  going  on  productive  paths.  The  absence  of  confirmatory  feedback  was  seen  by 
students  as  a  signal  that  an  error  had  occurred. 

Lepper  and  his  colleagues  have  also  considered  how  tutors  scaffold  students  learning, 
but  have  concentrated  on  the  motivational  aspects  of  tutorial  feedback  (Lepper  et  ah.  1990: 
Lepper  Sz  Chabav.  1988).  They  argued  that  a  major  goal  of  tutors  is  to  keep  students  from 
becoming  discouraged  and  from  blaming  themselves  when  problem  solving  difficulties  are 
encountered.  The  tutors  accomplished  this  in  two  ways.  First,  they  emphasized  that  the 
problems  were  hard,  thereby  redirecting  the  blame  from  the  students  to  the  problems,  and 
permitting  students  to  attribute  the  errors  to  the  difficulty  of  the  problems  rather  than 
to  a  lack  of  ability.  Second,  rather  than  telling  students  how  to  repair  errors.  Lepper  s 
tutors  asked  leading  questions  that  helped  students  identify  and  repair  errors  themselves. 
Similarly,  some  teachers  use  questions  and  counter-examples  to  help  students  uncover  faults 
in  their  own  reasoning  (Collins  k.  Stevens.  1982:  Collins.  Warnock,  &  Passafiume,  1975). 
These  analyses  suggest  that  tutors  keep  students  feeling  successful  by  allowing  them  to 
find  and  repair  errors,  thereby  feeling  in  control  of  the  problem  solving  (cf..  Scardamalia. 
Bereiter.  McLean.  Swallow,  <k  Woodruff.  1989),  and  to  blame  errors  on  external  factors. 
The  Fox  (1991)  and  Lepper  et  al.  (1990)  studies  suggest  that  much  of  the  work  in  tutoring 
sessions  is  performed  by  the  student,  even  after  errors.  The  tutor  takes  advantage  of 
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opportunities  to  help  students  remain  sure  of  themselves  and  their  problem  solving  success, 
and  to  ensure  that  students  notice  any  errors.  The  students  are  primarily  responsible  for 
repairing  errors,  but  the  tutor  will  scaffold  the  process  as  needed  by  asking  leading  questions 
and  providing  the  occasional^,  correct  answer. 

Putnam  (1987)  also  argued  that  tutors  are  primarily  interested  in  getting  students 
to  complete  the  material.  However.  Putnam  proposed  that  tutors  rely  not  primarily  on 
opportunistic  planning,  but  on  curriculum  scripts  that  suggest  a  loosely  ordered  set  of  tasks 
to  perform  during  a  session  to  guide  the  session.  These  curriculum  scripts  are  plans  that 
vary  little  across  sessions.  In  this  view,  errors  are  not  particularly  important  opportunities 
to  increase  student  understanding.  Instead,  tutors  try  to  get  students  back  on  to  a  correct 
track  by  giving  the  answer  to  the  problem. 

Much  as  Fox  (1991)  and  Lepper  et  al.  (1990)  focused  upon  the  active  role  of  the  students. 
Graesser  et  ah  (1993)  argued  that  tutoring  is  successful  because  sessions  are  structured  to 
allow  students  the  opportunity  to  learn  actively  through  their  own  questions.  Indeed, 
students  ask  approximately  100  times  more  questions  during  tutoring  than  in  classroom 
situations  (Kerry.  1987).  and  learning  through  asking  questions  may  be  superior  to  more 
passive  learning  (Graesser  et  ah.  1993).  Interestingly,  students  often  fail  to  understand 
questions  they  are  asked  or  the  answers  to  their  own  questions,  and  thus  tutors  must 
collaborate  with  students  to  clarify  the  meaning  of  questions  and  answers.  Graesser  (1993) 
argued  that  these  interactions  of  question,  answer,  and  collaborative  search  for  meaning 
form  a  five  step  script,  called  a  dialog  frame .  Dialog  frames  are  employed  throughout 
tutoring  sessions  to  guide  students'  knowledge  acquisition  and  problem  solving. 

Other  researchers  have  focused  on  the  role  of  errors  in  triggering  curriculum  scripts.  For 
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example,  Littman  et  al.  (1990)  provided  tutors  with  students  PASCAL  programs  and  asked 
them  to  plan  an  intervention.  They  found  that  the  tutors  structured  the  entire  interaction 
around  feedback  for  errors.  The  tutors  used  a  great  deal  of  domain  knowledge  about  the 
causes  and  severity  of  errors, to  decide  upon  an  order  for  remediating  them,  and  planned  to 
offer  very  directive  feedback  during  the  remediation  (Littman,  1991).  In  fact,  these  tutors 
used  tutorial  planning  schemas  based  upon  these  errors  that  guided  the  tutorial  sessions. 
Planning  schemas  arise  out  of  both  domain  knowledge  and  tutoring  knowledge,  and  capture, 
for  example,  the  fact  that  repairing  an  error  might  be  necessary  before  some  other  error 
could  be  examined,  or  several  errors  might  be  indicators  of  the  same  deep  confusion.  These 
schemas  allow  tutors  to  develop  an  optimal  structure  for  the  tutoring  session  that  maximizes 
the  success  of  student  repairs. 

McArthur  et  al.  (1990)  and  Schoenfeld.  Gamoran.  Kessel,  and  Leonard  (1992)  have  also 
argued  that  tutors  use  scripts  to  guide  their  behavior,  but  have  extended  their  analyses 
to  other  kinds  of  student  behaviors  besides  errors  as  triggers  for  a  script.  These  scripts, 
sometimes  called  tutorial  microplans  (McArthur  et  ah,  1990),  can  be  triggered  by  various 
actions,  including  errors,  new  problem  solving  goals,  and  pedagogical  goals.  Like  tutorial 
planning  schemas,  microplans  are  used  to  decide  how  to  respond  to  a  student  action,  with 
each  microplan  generating  one  or  many  tutorial  responses.  However,  because  microplans 
can  be  activated  bv  actions  other  than  errors,  they  offer  tutors  the  flexibility  to  respond 
to  students*  individual  needs  and  confusions  while  still  accomplishing  general  pedagogical 
goals.  For  example,  according  to  McArthur  et  al.  (1990),  tutors  often  remind  students  of 
what  they  are  doing  and  why  it  is  being  done,  thereby  keeping  them  aware  of  problem 
solving  goals. 
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This  brief  review  of  tutoring  has  revealed  two  foci  of  tutoring  research.  Some  researchers 
(e.g..  Fox.  1991:  Lepper  k  Chabav.  1988)  have  focused  upon  the  content  of  tutorial  utter¬ 
ances.  whereas  others  (e.g.,  Littman  et  ah.  1990:  McArthur  et  ah.  1990:  Putnam.  1987: 
Schoenfeld  et  ah,  1992)  hav$  largely  focused  upon  the  ways  that  tutors  organize  sessions. 
These  two  dimensions  are  essentially  independent  —  tutorial  scripts  do  not  necessarily 
specify  the  type  of  feedback  to  be  given. 

These  studies  have  revealed  a  range  of  tutorial  behaviors,  including  a  focus  on  positive 
outcomes  (Fox.  1991).  the  subtle  nature  of  tutorial  guidance  (Fox,  1991:  Lepper  et  ah.  1990). 
the  assistance  of  tutorial  guidance  in  helping  to  structure  the  problem  solving  (McArthur 
et  ah.  1990:  Putnam.  1987:  Schoenfeld  et  ah.  1992).  and  response  to  student  errors  (Littman 
et  ah,  1990).  Many  of  these  findings  reveal  the  active  role  of  students  in  problem  solving 
(Fox.  1991),  questioning  (Graesser  et  ah.  1993),  and  repairing  errors  (Fox.  1991:  Graesser 
et  ah,  1993).  Part  of  this  variation  in  these  tutorial  behaviors  is  presumably  due  to  varia¬ 
tions  in  the  problem  solving  context.  These  studies  varied  in  factors  such  as  the  age  of  the 
students  and  the  tutors,  whether  the  session  was  remedial  or  was  covering  the  material  for 
the  first  time,  whether  the  students'  participation  was  voluntary  or  required,  and  so  on. 

In  general,  these  tutoring  studies  have  examined  portions  of  tutoring  sessions  and  have 
characterized  certain  interactions  of  theoretical  interest.  Although  these  snapshots  of  tutor¬ 
ing  have  cast  a  great  deal  of  light  on  the  tutorial  process,  the  complete  picture  of  tutoring 
has  yet  to  be  developed.  Developing  a  model  of  tutoring  that  characterizes  the  ways  in 
which  tutorial  assistance  leads  to  pedagogical  success  requires  examining  tutoring  over  a 
long  period  of  time  with  a  variety  of  students  to  examine  the  various  contexts  in  which 
different  tutorial  behaviors  may  arise.  Presumably,  all  of  the  above  theories  describe  differ- 
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ent  aspects  of  the  range  of  tutorial  behavior.  Previous  worK  has  not  yet  analyzed  complete 
tutorial  sessions  with  the  aim  of  characterizing  the  contexts  giving  rise  to  the  full  range  of 
tutorial  assistance. 

The  goal  of  the  study  presented  here  is  to  describe  the  situations  that  lead  tutors  to 
behave  in  the  ways  we  have  just  reviewed.  We  will  argue  that  tutors  guide  problem  solving 
in  two  principal  manners  and  that  the  situational  characteristics  affect  the  manner  chosen 
and  the  formation  of  the  guidance.  First,  they  offer  rapid  and  explicit  feedback  to  student 
actions  that  tells  the  student  whether  or  not  the  action  was  correct.  Second,  in  the  event 
of  a  mistake,  students  and  tutors  collaborate  in  repairing  the  error.  Tutors  carefully  choose 
feedback  to  allow  students  to  perform  many  components  of  the  error  recovery  process 
(Merrill  et  ah.  1992).  Furthermore,  we  will  use  these  behavioral  findings  along  with  other 
problem  solving  research  to  offer  a  potential  explanation  for  the  success  of  tutoring. 

To  investigate  these  issues,  we  designed  a  controlled  learning  task,  in  which  computer 
novices  learned  basic  programming  concepts  with  the  assistance  of  a  tutor.  The  data  pre¬ 
sented  here  were  gathered  in  an  experiment  contrasting  students  working  a  preset  curricula 
of  LISP  programming  problems  in  four  different  learning  environments.  In  this  paper,  we 
present  data  from  two  of  the  four  conditions.  The  first  condition  was  a  traditional  one- 
on-one  tutoring  situation,  in  which  students  solved  LISP  programming  problems  with  the 
constant  assistance  of  a  tutor.  To  allow  us  to  explore  tutors  means  of  guidance,  we  audio- 
taped  all  verbal  interactions  between  student  and  tutor.  We  used  two  different  tutors,  each 
with  significant  tutoring  experience,  to  increase  the  diversity  of  behaviors  that  would  be 
revealed  by  the  discourse  analysis  of  the  tutorial  interactions.  The  goal  of  this  analysis  is  to 
characterize  tutorial  actions  in  long-term  interventions  that  cover  a  wide  range  of  material. 
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Thus,  we  chose  to  emphasize  depth  of  interaction  with  each  tutor  rather  than  number  of 
tutors. 

As  we  have  noted,  it  is  plausible  that  many  factors  affect  tutorial  behavior,  such  as  the 
domain  being  studied,  the  age  of  the  students,  and  whether  the  session  is  remedial  or  not. 
This  study  uses  non-remedial  sessions  and  college-aged  students,  and  thus  examines  one 
possible  subset  of  the  space  of  tutorial  possibilities.  We  focused  on  students  learning  new 
material  because  these  sessions  are  likely  to  encompass  a  range  of  problem  solving  scenarios, 
including  those  in  which  the  student  succeeds  and  those  in  which  the  student  encounters 
obstacles. 

The  control  condition  for  this  paper,  called  the  independent  problem  solving  condition, 
consisted  of  novices  covering  the  same  material  and  solving  the  same  problems,  but  without 
tutorial  assistance.  Verbalizations  were  not  recorded  in  this  condition. 

This  study  includes  approximately  50  hours  of  student-tutor  verbal  interactions.  To  an¬ 
alyze  these  data,  we  used  a  style  of  discourse  analysis  similar  to  protocol  analysis  (Ericsson 

Simon.  1984),  to  examine  the  contexts  in  which  different  tutorial  actions  arise  and  the 
outcomes  of  these  actions.  Our  version  of  this  technique  is  quite  similar  to  discourse  anal¬ 
ysis  in  that  it  analyzes  the  language  of  two  or  more  people  talking  during  problem  solving 
to  reveal  the  actions  employed  during  these  dialogues.  The  important  theoretical  claim  of 
protocol  analysis  and  of  our  approach  to  discourse  analysis  is  that  the  researcher  cannot 
presume  to  have  full  access  to  the  mental  states  of  the  problem  solvers,  and  so  must  focus 
solely  upon  that  information  that  is  completely  explicit  in  the  participants'  utterances. 

Bv  looking  at  patterns  of  utterances,  a  researcher  attains  limited  access  to  the  mental 
processes  made  while  solving  a  problem  based  only  on  the  information  to  which  the  subject 
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is  attending  as  well  as  any  inferences,  assertions,  or  questions  made  explicit  by  the  subject. 
The  researcher  develops  categories  based  upon  the  explicit  content  of  each  utterance,  not 
what  the  researcher  believes  the  speaker  meant,  and  categorizes  all  utterances  made  during 
the  task  (Bakeman  Gottrqan.  1986).  These  categorizations  specif}’  tne  various  problem 
solving  events  that  occur  on  the  way  to  a  solution.  This  technique  allows  us  to  look 
for  contingencies  between  problem  solving  context  and  tutorial  action  by  examining  the 
transitions  from  one  sort  of  event  to  another. 

We  developed  a  coding  scheme  containing  36  categories  designed  to  capture  the  full 
range  of  both  student  and  tutor  behaviors  during  problem  solving.  For  example,  we  had 
categories  for  utterances  such  as  a  student  asking  for  help,  setting  a  goal,  or  generating  a 
concrete  example,  as  well  as  for  tutorial  actions  such  as  error  feedback  or  goal  setting  (the 
complete  scheme  is  described  below).  We  categorized  each  and  every  utterance  made  by 
tutor  or  student  during  the  approximately  50  hours  of  sessions.  Thus,  this  study  presents 
a  fine-grained  picture  of  tutoring  over  an  extended  period  of  problem  solving. 

These  extended  microanalvses  enable  us  to  examine  tutorial  behaviors  across  many  dif¬ 
ferent  contexts  within  each  student  and  with  different  students  to  determine  which  aspects 
of  the  behaviors  are  components  of  successful  tutoring  in  this  type  of  domain.  In  sum. 
this  study  enables  us  to  characterize  the  ways  tutors  assist  problem  solving  in  procedural 
domains  and  to  develop  a  model  to  show  how  these  behaviors  make  tutors  so  successful. 
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Method 

Subjects 

The  subjects  in  the  overall  experiment  were  16  Princeton  University  undergraduates 
and  graduate  students  recruited  through  sign-up  sheets  on  campus  (eight  in  the  one-on- 
one  tutoring  condition  and  eight  in  the  independent  problem  solving  condition).  Students 
were  randomly  assigned  to  conditions,  and  were  paid  S5.00  per  hour  tor  their  participation. 
The  participants  included  an  equal  number  of  males  and  females,  all  with  no  previous 
programming  experience.  To  minimize  individual  differences  across  conditions,  gender  and 
Math  SAT.  a  good  predictor  of  success  in  learning  to  program  (Mayer,  Dyck,  k  Vilberg. 
1986).  were  roughly  balanced  across  conditions.  The  overall  mean  Math  SAT  of  the  subjects 
was  690. 

Students  in  the  one-on-one  tutoring  condition  were  matched  with  a  tutor  of  the  same 
gender.  Two  Princeton  University  undergraduates  acted  as  tutors  in  this  experiment.  The 
female  tutor  had  previous  experience  tutoring  math  and  science  in  high  school,  and  was  an 
experienced  LISP  programmer.  The  male  tutor  had  experience  teaching  LOGO  to  students 
in  summer  camps;  he  was  also  an  experienced  LISP  programmer.  Both  tutors  were  unaware 
of  the  goals  of  the  study. 

Materials 

The  students  worked  56  problems  interspersed  throughout  the  first  three  chapters  of 
an  introductory  LISP  textbook.  Essential  LISP  (Anderson.  Corbett,  k  Reiser.  1987 j .  The 
three  chapters  amounted  to  roughly  50  pages  of  text,  and  introduced  25  built-in  LISP 
functions,  variables  and  constants,  the  form  of  basic  function  definitions,  and  the  use  or 
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conditionals. 

We  constructed  two  cumulative  posttests  that  covered  material  in  the  second  and  third 
chapters.  These  pencil-and-paper  posttests  consisted  of  problems  requiring  students  to 
generate  LISP  programs  to  £olve  small  problems,  to  find  and  repair  errors  in  previously 
generated  programs,  and  to  give  the  output  of  LISP  functions  with  given  input  values. 

The  students  were  able  to  work  at  all  times  during  the  learning  session  on  a  computer 
terminal  running  a  LISP  interpreter  that  had  been  modified  to  store  and  timestamp  all 
keystrokes  the  students  made.  The  interpreter  did  not  contain  the  traditional  LISP  debug¬ 
ging  mode,  which  often  confuses  novices.  There  was  also  a  simple  screen  editor  available 
for  the  students  to  use  to  edit  function  definitions. 

Procedure 

Students  were  told  to  read  the  material  in  the  textbook  and  to  attempt  to  solve  the 
problem  sets  intermixed  in  the  chapters.  The  students  received  a  demonstration  of  the 
computer  system  at  the  beginning  of  the  first  session  and  a  demonstration  of  the  editing 
facilities  at  the  beginning  of  the  second  session.  In  addition  to  the  56  assigned  problems, 
all  students  took  the  untimed  posttests  after  the  second  and  third  chapters.  Students  were 
allowed  to  work  at  their  own  pace,  and  took  between  5  and  10  hours  to  complete  the  task, 
distributed  over  three  to  five  days.  They  were  free  to  refer  to  the  text  at  any  time.  All 
students  completed  ail  problems  correctly. 

One-on-one  tutoring:  Subjects  in  the  one-on-one  tutoring  condition  worked  through  the 
material  with  the  assistance  of  an  experienced  human  tutor.  The  tutors  were  instructed  to 
use  the  textbook  and  all  of  the  56  assigned  problems,  but  were  not  told  to  use  a  particular 
method  of  tutoring  with  the  students:  instead,  they  were  to  rely  upon  their  tutoring  ex- 
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pertise.  The  student  and  tutor  were  seated  side  by  side  at  a  table  containing  the  computer 
terminal  and  a  tape  recorder.  The  keyboard  was  placed  in  front  of  the  student  to  facilitate 
the  students'  typing,  but  the  tutors  could  type,  if  needed.  The  tutors  were  instructed  to 
require  the  students  to  solve^each  problem  correctly  before  moving  on  to  the  next  problem 
in  the  pre-set  sequence. 

Independent  problem  solving:  In  the  independent  problem  solving  condition,  students 
worked  through  the  problems  without  access  to  a  tutor.  The  experimenter  checked  the 
solutions  after  each  chapter  and  told  the  students  which  problems,  if  any.  were  incorrect. 
The  experimenter  did  not  convey  anything,  about  the  errors  in  the  solution,  but  simply 
reported  that  the  solution  was  incorrect.  The  students  were  then  required  to  make  the 
necessary  repairs.  Thus,  subjects  in  this  condition  were  also  required  to  solve  all  56  assigned 
problems  correctly.  The  students  were  allowed  to  ask  the  experimenter  questions  if  they 
felt  completely  confused.  In  these  infrequent  cases,  the  experimenter  would  offer  some 
small  amount  of  assistance,  such  as  pointing  the  student  back  to  the  relevant  section  of  the 
textbook. 

Discourse  Analysis  Methods 

To  analyze  the  discourse  between  tutors  and  students,  we  first  transcribed  the  complete 
protocols  from  all  eight  student-tutor  pairs.  Then,  we  interspersed  the  records  made  by  the 
computer  of  all  LISP  interactions  into  the  transcriptions  to  provide  one  complete  trace  of 
all  verbal  behavior  and  interactions  wdth  the  computer.  This  complete  trace  serves  as  the 
data  for  this  analysis.  The  goal  of  this  analysis  is  to  uncover  the  behaviors  giving  rise  to 
tutorial  effectiveness  and  the  situations  in  which  these  behaviors  occur  by  examining  the 
patterns  of  student-tutor  utterances  throughout  the  problem  solving. 
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Before  discourse  analysis  was  performed,  all  transcripts  of  verbalizations  must  be  divided 
into  smaller  units  corresponding  to  codeable  events.  Then,  each  segment  was  categorized 
according  to  the  type  of  student  or  tutorial  action.  This  process  is  known  as  segmentation 
(Bakeman  &  Gottman.  1986^..  When  dividing  a  protocol  into  segments,  the  segmenter  must 
make  a  decision  about  when  one  segment  ends  and  another  begins.  Explicit  segmentation 
rules  are  used  to  ensure  reliable  segmentation,  typically  by  requiring  segmenters  to  make 
few  inferences  (Bakeman  k  Gottman,  1986).  One  way  to  measure  the  success  of  these 
rules  is  to  measure  percent  agreement,  capturing  the  extent  to  which  segmenters  divide  the 
protocol  similarly. 

In  this  study,  we  used  a  method  for  breaking  the  discourse  into  events  we  call  segmenta¬ 
tion  by  idea ,  based  upon  identifying  when  a  speaker  is  discussing  a  new  point.  All  discourse 
on  any  one  point  by  one  speaker  becomes  one  segment.  Thus,  each  time  a  new  idea  is 
entered  into  the  discourse  a  new  segment  is  created,  allowing  detailed  access  to  each  and 
every  topic  of  any  speaker's  discourse. 

Segmentation  by  idea  differs  from  turn-taking  segmentation  schemes,  a  very  common 
scheme  (e.g.,  Bloom.  Rocissano.  k  Hood.  1976).  in  that  turn-taking  schemes  typically 
attempt  to  take  the  whole  utterance  of  a  speaker  (one  turn)  as  the  unit  of  analysis  to  be 
categorized,  whereas  segmentation  by  idea  allows  a  single  utterance  to  be  broken  up  if  it 
expresses  multiple  points. 

Insert  Table  2  about  here 

For  example,  consider  the  protocol  shown  in  Table  2.  which  is  both  segmented  and 
categorized.  In  this  part  of  the  session,  the  tutor  is  explaining  how  LISP  matches  parameters 
in  a  student’s  function  to  actual  values  when  that  function  is  called.  Note  that  the  a  single 
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tutorial  utterance  was  divided  into  three  categorizable  segments  (events  9-11  in  Table  2) 
whereas  it  would  have  been  classified  as  a  single  turn  in  a  turn-taking  scheme.  Segmentation 
by  idea  allows  consideration  of  the  individual  problem  solving  events  that  occur  in  a  series 
within  one  participants  condiments  rather  than  considering  them  together.  Furthermore, 
segmentation  by  idea  differs  from  other  methods  of  segmentation  in  that  a  segment  can 
continue  across  an  utterance  of  the  other  person  if  the  speaker  fails  to  acknowledge  the 
second  person's  speech  in  any  way.  For  example,  consider  event  number  5  in  Table  2.  Here 
the  tutor  does  not  verbally  acknowledge  any  of  the  student  s  comments,  and  continues 
describing  the  same  concept.  Thus,  this  entire  interaction  was  segmented  as  one  event. 
This  method  thus  allows  each  segment  to  capture  a  single  complete  problem  solving  event. 

To  ensure  the  reliability  of  our  scheme,  two  of  the  authors  independently  segmented  all 
of  the  protocols,  and  differences  between  their  segmentations  were  resolved  between  them. 
Instructions  for  segmentation  are  shown  in  Appendix  A.  The  two  segmenters  initially  agreed 
on  98%  of  segments,  calculated  across  all  protocols,  indicating  that  the  rules  we  used  could 
be  implemented  very  reliably  (Bakeman  Gottman.  1986)  and  that  there  were  very  few 
differences  to  resolve.  This  study  analyzes  approximately  15,000  segments. 

After  segmenting  all  protocols,  we  next  assigned  each  segment  to  one  of  36  categories 
that  captured  each  student  or  tutor  action.  We  developed  these  categories  to  represent 
the  information  expressed  throughout  the  problem  solving.  They  allow  us  to  specify  the 
problem  solving  contexts  that  lead  to  the  various  tutorial  behaviors  and  to  examine  the 
pedagogical  strategies  of  the  tutors.  We  designed  categories  to  capture  tutorial  behaviors 
highlighted  as  crucial  for  pedagogical  success  in  the  theories  of  tutoring  presented  earlier. 
For  example,  we  created  categories  for  tutor  confirmatory  feedback  (Fox,  1991).  tutor  moti- 
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rational  feedback  (Lepper  et  al..  1990).  and  tutor  goal  remindings  (McArthur  et  ah.  1990). 
We  also  designed  categories  for  events  viewed  as  important  to  learning  in  general,  such  as 
making  assertions,  trying  solutions,  setting  goals,  offering  explanations,  and  so  forth.  A 
complete  list  of  the  categories  of  student  events  is  presented  in  Table  3;  Table  4  contains 
the  tutor  events.  Definitions  and  examples  of  each  category  along  with  the  number  of  oc¬ 
currences  of  each  are  included  in  Appendix  B.  Each  category  was  designed  so  an  utterance 
could  be  classified  by  the  explicit  meaning  of  the  utterance. 

Insert  Table  3  about  here 


Insert  Table  4  about  here 

In  general  terms,  the  student  categories  shown  in  Table  3  capture  actions  such  as  prob¬ 
lem  solving  actions,  asking  for  tutorial  help,  indicating  understanding  of  tutor  utterances, 
checking  the  current  answer,  and  miscellaneous  comments.  Similarly,  the  tutor  categories 
displayed  in  Table  4  consist  of  groups  of  actions  such  as  when  the  tutor  performs  a  portion 
of  the  problem  solving,  offers  guidance  to  the  student  about  ongoing  problem  solving,  offers 
confirmatory  feedback,  gives  error  feedback,  assesses  the  student’s  understanding  of  a  topic, 
helps  the  student  check  the  answer,  and  makes  miscellaneous  off-task  comments. 

The  coding  scheme  was  designed  to  be  dependent  upon  the  content  of  the  utterance 
rather  than  upon  the  form  of  the  speech  act  used.  Although  the  choice  of  speech  act  could 
affect  the  way  a  student  responds  to  an  utterance  (Graesser  et  ah,  1993:  Lepper  et  ah. 
1990).  we  viewed  the  content  of  the  utterance  as  most  representative  of  the  problem  solving 
state  of  the  student  and  tutor.  For  our  analyses  of  tutorial  responses  to  problem  solving 
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situations,  we  wanted  to  focus  on  the  information  communicated  rather  than  upon  the  style 
with  which  it  is  expressed.  This  information  defines  the  situations  to  which  the  tutors  are 
responding.  Thus,  utterances  1  and  2  below  are  both  categorized  as  Tutor  Corrections . 
because  each  conveys  the  sqme  information,  namely  that  the  student  needs  a  quotation 
mark  in  the  program. 

1.  Tutor:  No  —  put  a  quote  there. 

2.  Tutor:  Don't  you  need  a  quote  there? 

We  categorized  each  segment  into  one  and  only  one  category,  because  more  power¬ 
ful  analyses  are  possible  for  mutually  exclusive  categories  (Bakeman  &  Gottman,  1986). 
Table  2  displays  the  interaction  after  both  segmentation  and  categorization.  Initially  in 
Table  2.  the  tutor  points  the  student  to  an  example  in  the  text.  This  is  a  Tutor  Focus 
Attention .  The  student  responds  affirmatively  to  this  point,  offering  a  Student  Confirma¬ 
tion.  Soon  thereafter,  the  tutor  works  through  the  solution  as  the  computer  would,  which 
falls  into  the  categorv  TSP,  Tutor  Simulate  Process.  After  this  TSP,  the  student  offers  a 
confirmation,  and  the  tutor  begins  a  discussion  of  how  LISP  treats  function  parameters. 

All  protocols  were  independently  categorized  by  two  of  the  authors.  The  rules  followed 
by  the  two  coders  are  presented  in  Appendix  A.  After  completing  all  categorization,  we 
examined  the  extent  to  which  the  coders  categorized  events  in  a  similar  manner.  Cohen  s 
Kappa  (k)  (Bakeman  Sz  Gottman.  1986:  Cohen.  1960)  is  often  used  to  measure  reliability 
in  categorization.  Kappa  captures  the  agreement  among  multiple  coders  but  adjusts  the 
resulting  value  for  the  amount  of  agreement  between  coders  that  would  be  expected  due 
to  chance.  A  value  of  Kappa  greater  than  0.70  is  considered  indicative  of  a  reliable  coding 
scheme  (Bakeman  Gottman.  1986).  The  categorization  in  this  study  was  quite  reliable. 
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k  =  0.81. 

Our  categorization  scheme  based  upon  segmentation  by  idea  does  have  potential  limi¬ 
tations.  For  example,  recall  that  each  utterance  is  categorized  as  one  event.  However,  it 
seems  possible  that  an  utterance  could  in  fact  serve  several  conversational  goals.  For  exam¬ 
ple,  an  utterance  classified  as  a  Tutor  Elaboration  could  also  contain  a  Tutor  Confirmation: 
“Yes,  right,  you  remembered  that  LAST  returns  a  list.'  Since  there  is  no  explicit  evidence 
of  a  change  of  topic,  these  would  be  segmented  together  and  coded  as  one  utterance,  even 
though  the  utterance  appears  to  serve  two  pedagogical  goals  simultaneously  to  offer  a 
confirmation  and  to  elaborate  upon  information  present  in  the  solution.  In  the  analysis 
presented  below,  we  will  take  the  overlapping  goals  of  different  categories  into  account  by 
merging  certain  categories  that  overlap  significantly. 

This  limitation  did  not  appear  to  interfere  with  categorization.  The  high  reliability  of 
the  categorization  indicates  that  it  was  in  fact  possible  to  assign  each  utterance  to  a  single 
category  with  high  reliability.  Thus,  constraining  each  utterance  to  only  one  category 
appears  to  be  a  reasonable  principle  for  categorizing  these  problem  solving  events. 

In  addition  to  identifying  each  problem  solving  event,  we  also  identified  those  actions 
that  contained  an  erroneous  assertion  or  solution  component.  Errors  play  a  critical  role  in 
learning.  For  example,  explaining  why  an  error  occurred  may  help  novices  avoid  the  error  in 
the  future  and  highlight  areas  of  confusion  (Chi,  Bassok.  Lewis,  Reimann.  &  Glaser,  1989: 
Schank  &  Leake.  1989).  However,  errors  can  also  lead  to  serious  floundering,  potentially 
interfering  with  learning  (Anderson,  Boyle,  h  Reiser.  1985:  Lewis  Anderson.  1985;  Reiser. 
Beekelaar.  Tyle.  &:  Merrill.  1991).  Tutorial  assistance  provided  for  locating  and  repairing 
errors  has  become  a  focus  of  research  in  both  human  and  computer-based  tutoring  (e.g.. 
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Anderson  et  al..  1985:  Littman  et  al..  1990:  McArthur  et  ah.  1990:  Reiser.  Kimberg.  Lovett, 
ik  Rannev.  1992).  Thus,  errors  represent  particularly  critical  events  around  which  to  focus 
our  analvsis  of  tutorial  strategies  and  the  associated  learning  outcomes.  To  achieve  this 
goal,  we  located  and  categorized  each  and  every  error,  and  then  identified  the  utterance 
that  indicated  an  error  had  occurred,  according  to  the  following  procedures. 

Again,  two  of  the  authors  independently  looked  through  all  the  utterances  and  computer 
interactions  to  locate  student  errors.  Errors  included  student  goals  that  were  not  needed 
in  the  problem,  required  goals  that  were  forgotten,  incorrect  assertions  about  functions 
or  concepts,  and  syntactic  mistakes,  as  well  as  slips  such  as  typographical  mistakes.  In 
contrast  to  the  categorization  of  conversational  events,  in  this  analysis  we  considered  the 
speech  act  of  the  utterance.  We  did  not  categorize  questions  as  errors,  because  questions  are 
explicit  requests  for  information,  rather  than  situations  where  a  student  asserts  something 
to  be  true  that  is  false.  Thus  utterance  3  was  marked  as  an  error  because  the  assertion 
about  append  is  false,  but  utterance  4  was  not  categorized  as  an  error  even  though  the  fact 
proposed  is  incorrect. 

3.  OK,  append .  append .  let's  see.  that's  the  one  that  takes  an  atom  and  a  list. . . 

4.  Is  append  the  one  that  takes  two  atoms? 

Focusing  on  non-question  mistakes  allows  us  to  see  precisely  what  tutors  do  when  students 
make  a  mistake  rather  than  what  happens  following  an  explicit  request  for  information.  We 
were  able  to  identifv  the  errors  reliably,  k  =  0.75.  There  were  1.242  errors  in  the  problem 
solving  sessions,  or  approximately  25  per  hour.  More  errors  occurred  during  the  third 
chapter,  as  the  material  became  more  difficult,  but  substantial  numbers  of  errors  occurred 
during  the  first  and  second  chapters  as  well. 
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After  identifying  the  erroneous  actions,  we  categorized  each  one.  This  categorization 
of  errors  is  designed  to  determine  if  tutors  respond  differently  to  errors  of  varying  tvpes. 
Accordingly,  our  error  categories  captured  mistakes  that  have  been  discussed  as  central 
for  learning,  such  as  errors  ip  the  syntax  of  solutions  (Anderson  et  ah,  1985,  Reiser  et  ah, 

1991) ,  confusions  about  the  semantics  of  basic  operators  (Anderson.  1989;  Reiser  et  ah, 

1992) ,  problems  with  goal  structures  (McArthur  et  ah.  1990:  Singley,  1990;  Soloway.  1986; 
VanLehn,  1990),  and  other  errors  that  one  would  expect  to  occur  in  such  a  task,  such  as 
typographical  errors.  We  originally  designed  nine  error  categories,  including  four  slightly 
differing  variants  of  one  error  group  called  semantic  errors.  In  fact,  we  found  occurrences 
of  only  two  of  these  four  variants  among  the  student  errors,  so  we  discarded  the  two  empty 
categories.  We  also  initially  considered  dead  code,  a  situation  in  which  extra  functions 
are  left  in  a  solution  but  do  not  affect  it  in  any  way,  as  potential  errors.  However,  the 
tutors  never  responded  to  these  situations,  and  the  students  solutions  actually  produced 
the  correct  result,  so  we  did  not  consider  these  18  cases  in  our  analyses  of  student  errors. 
The  remaining  six  error  categories  shown  in  Table  5  included  1,224  errors.  Once  again,  two 
authors  independently  categorized  the  errors.  The  categorization  was  reliable,  k  —  .70. 

Insert  Table  5  about  here 

Finally,  after  locating  the  errors,  we  looked  to  see  which  participant  indicated  that  an 
error  had  occurred  and  how  the  indication  was  performed.  For  example,  the  student  might 
make  an  error  and  then  notice  it  and  begin  a  repair.  In  example  5.  the  student  makes  a 
syntactic  error  in  the  else  clause  of  a  conditional  by  putting  two  parentheses  instead  of  just 
one  before  the  b  and  then  flags  the  error  herself. 
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5.  Student:  [typing] 

(defun  classify  (arg) 

(cond  ((numberp  arg)  'number! 

((null  arg)  ’nil)  < 

((t 

Student:  Whoops,  I  don't  need  that  many. 

Tutor:  Right,  exactly.  You  caught  yourself. 

Alternatively,  the  tutor  might  comment  on  the  error,  as  in  example  6,  which  occurred  earlier 
in  the  same  problem. 

6.  Student:  (typing] 

(defun  classify  (arg) 

(cond  ((numberp  arg)  number 

Tutor:  Now  actually,  um,  for  number,  you  want  the  actual  word. 

Student:  So  I  have  to  put  this  [a  quotation  mark]. 

Tutor:  Yes.  you  have  to  put  a  quote. 

Student:  [typing] 

<DEL>. . .  <DEL> 'number) 

The  students  self-initiated  correction  in  example  5  and  the  tutor  s  comment  in  exam¬ 
ple  6  indicated  that  an  error  had  occurred.  We  call  such  utterances  error  flags .  We  classified 
an  utterance  a 5  an  error  flag  if  it  indicated  that  a  problem  solving  event  was  incorrect.  Error 
flags  ranged  from  very  specific,  as  in  example  6.  to  very  general,  such  as  the  tutor  saying 
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"Look  back  up  there  —  there  might  be  a  problem."  General  utterances  alerted  the  student 
that  one  of  the  recent  steps  contained  an  error,  but  did  not  tell  which  step  was  wrong. 

We  did  not  restrict  the  categories  of  utterances  that  could  serve  as  error  flags.  Of  course, 
given  the  nature  of  some  of  {.he  category  definitions,  certain  categories,  like  Tutor  Confirm 
Step,  were  not  used  to  indicate  that  an  error  had  occurred.  Thus,  although  all  categories 
could  serve  as  possible  flags,  only  a  subset  were  actually  used  as  error  flags.  Assist  Plan 
Assertion  was  the  most  common  error  flag  from  the  student.  Tutor  Error  Feedback  was  the 
most  common  tutor  error  flag,  followed  by  Tutor  Prompt.  Two  of  the  authors  marked  the 
flag  for  each  of  the  1224  errors  reliably,  k  =  .75. 

Errors  did  not  always  result  in  an  incorrect  solution  attempt.  In  some  cases,  students 
made  an  error  in  a  step  but  immediately  located  it  and  began  the  repair  within  the  same 
event,  as  in  example  7. 

7.  Student:  [typing]  (mv-or  a  <DEL><DEL>'a 

This  student  did  not  put  a  required  quotation  mark  before  the  constant  a,  but  immediately 
repaired  the  error  without  assistance.  We  marked  this  as  an  error  with  an  immediate  flag 
by  the  student.  Even  though  the  student  needed  no  help  in  this  case,  she  experienced  an 
impasse  that  had  to  be  overcome.  To  account  for  all  impasses  and  their  repairs,  we  included 
these  cases  in  the  analyses  as  well.  Furthermore,  in  some  of  these  cases  in  which  the  student 
repaired  his  or  her  own  error,  there  is  no  explicit  utterance  that  marks  the  error.  Instead, 
the  self-correction  behavior  is  considered  both  to  flag  and  to  repair  the  error. 

In  many  cases,  the  error  flag  only  initiated  the  error  repair  process,  which  could  require 
several  events  to  complete.  To  determine  how  many  events  were  required  to  repair  errors, 
two  of  the  authors  independently  located  the  event  that  achieved  the  repair  for  each  error. 
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The  repair  might  occur  with  the  same  event,  as  in  example  7,  or  might  occur  a  few  events 
later.  Finding  the  repair  location,  given  the  location  of  the  error  itself,  is  very  reliable. 
k  =  0.90. 

Having  described  the  categorization  of  each  and  every  utterance,  the  finding  of  all  errors, 
the  identification  of  the  utterance  that  begins  the  error  recovery  process,  and  the  location 
of  the  end  of  each  error  episode,  we  next  turn  to  the  analyses  of  these  data. 

Results  and  Discussion 

Before  describing  the  tutors*  methods  of  assisting  the  students  during  the  learning 
sessions,  we  must  demonstrate  that  the  tutors  were  in  fact  effective.  To  do  so.  we  analyzed 
the  students'  posttests  and  the  duration  of  the  learning  sessions.  Recall  that  all  students, 
regardless  of  condition,  worked  on  all  the  problems  until  they  were  all  correctly  solved.  The 
tutored  students  completed  the  material  in  just  over  half  the  time  that  the  non-tutored 
students  required,  300  mins  versus  550  mins.  F(l,  13)  =  24.5, p  <  .01.  There  were  no 
differences  between  the  groups*  performance  on  the  posttest.  Students  in  the  one-on-one 
tutoring  condition  received  97%  of  the  points  possible,  and  the  independent  problem  solving- 
students  scored  95%  on  average.  This  lack  of  posttest  differences  does  not  indicate  a  lack  of 
pedagogical  effectiveness  for  tutors,  because  one  would  expect  that  solving  all  the  problems 
correctly  should  lead  to  significant  domain  mastery  (Anderson  Sz  Corbett.  1993:  Newell, 
1990).  Thus,  it  is  not  surprising  that  both  groups  were  able  to  achieve  the  same  degree 
of  understanding  of  LISP.  The  important  difference  is  the  time  it  took  to  do  so  —  the 
tutored  students  achieved  equivalent  domain  mastery,  in  spite  of  spending  substantially 
less  time  on  task  than  the  independent  problem  solving  students.  Thus,  the  tutors  did 
provide  clear  cognitive  benefits,  and  so  we  next  examine  the  protocol  data  to  specify  the 
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actions  performed  by  the  tutors  and  the  situations  in  which  they  occurred. 

In  this  section,  we  present  our  analyses  of  the  approximately  15.000  student-tutor  in¬ 
teractions  over  the  50  hours  of  sessions  to  describe  the  ways  tutors  assist  students  in  devel¬ 
oping  domain  mastery.  We  present  a  model  to  show  why  the  assistance  should  be  helpful. 
Tables  1-2  demonstrated  the  complexity  of  the  student-tutor  interactions,  including  con¬ 
firmations,  corrections,  goals,  and  much  else.  Our  fine-grained  identification  of  all  student 
actions  and  tutorial  responses  in  extended  problem  solving  sessions  allows  us  to  investigate 
how  tutors  support  and  guide  students'  problem  solving. 

Domain  mastery  typically  begins  by  studying  expository  text  and  annotated  examples 
(Chi  et  al.,  1989;  Faries.  1991:  Gentner.  1983;  Gick  k  Holyoak.  1980:  Pirolli,  1991;  VanLehn. 
Jones,  k  Chi,  1992).  Elaboration  of  declarative  material  via  questioning,  predicting,  and 
explaining  can  facilitate  solving  problems  later  (Chi  et  ah,  1989:  Graesser,  1992).  Another 
critical  location  of  learning  is  the  attempted  application  of  the  declarative  knowledge  gained 
from  the  text  and  examples  to  solve  new  problems  (Anderson.  1983.  1987;  Trafton  k  Reiser. 
1993a).  We  are  concerned  with  the  overall  development  of  domain  expertise  in  these  learning 
sessions  rather  than  the  differential  roles  of  solving  problems  and  understanding  expository 
materials.  Thus,  we  consider  student  events  occurring  either  while  reading  or  working 
on  assigned  exercises  as  problem  solving  actions,  and  focus  on  the  role  of  these  events  in 
learning. 

Protocol  analyses  typically  make  use  of  new  categories  combined  out  of  the  originally 
coded  events  (Bakeman  k  Gottman.  1986).  Often  the  original  categories  are  coded  at 
an  extremely  fine  grain,  and  then  clusters  of  events  taken  together  represent  functional 
groups.  For  example,  several  of  the  original  student  events  taken  together  describe  the 
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process  of  problem  solving,  upon  which  we  focus.  These  student  events  capture  different 
actions  that  could  take  place  during  problem  solving.  In  our  study,  these  problem  solving 
actions  included  assertions  and  elaborations  made  while  reading  the  text,  generating  or 
studying  a  concrete  example^  setting  goals,  or  creating  new  LISP  expressions.  All  of  these 
are  categories  in  our  coding  scheme.  However,  the  fine  distinctions  between  them  are  not 
relevant  for  understanding  how  the  tutors  assist  overall  problem  solving,  including  encoding 
the  text  and  examples.  Thus,  we  combined  these  categories  into  a  new  category  called 
Student  Problem  Solving  Action  (SPSA).  which  will  be  used  throughout  the  next  analyses. 

In  addition,  we  created  another  higher  level  category.  Tutor  Confirm  Step  (TCS).  Recall 
that  earlier,  we  pointed  out  the  segmentation  by  idea  often  led  to  Tutor  Elaborations  includ¬ 
ing  explicit  confirmations.  Since  all  segmentation  was  performed  before  categorizing  any 
utterance,  we  could  not  go  back  and  break  the  confirmatory  portion  out  of  the  elaborative 
portion  of  a  Tutor  Elaboration  that  contained  an  confirmation.  Thus,  we  decided  to  include 
Tutor  Elaborations  containing  explicit  confirmations  in  TCS.  Furthermore,  elaborating  on 
a  student  assertion  usually  implies  an  implicit  confirmation,  since  the  new  information  is 
added  to  the  tacitly  agreed  to  student  action.  Therefore,  we  defined  the  category  Tutor 
Confirm  Step  to  include  utterances  originally  categorized  as  Tutor  Confirmations  as  well 
as  utterances  originally  coded  as  Tutor  Elaborations.  To  focus  on  the  role  of  confirmatory 
feedback  in  problem  solving,  we  joined  these  two  categories  together  into  the  new  Tutor 
Confirm  Step. 

Insert  Figure  1  about  here 

Figure  1  shows  the  events  in  the  tutoring  sessions,  displaying  the  results  of  our  catego¬ 
rization  of  the  data.  Each  object  in  Figure  1  is  a  type  of  event.  The  half  circles  are  tutor 
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events  such  as  Tutor  Confirm  Step,  in  which  the  tutor  offered  confirmatory  feedback  to  the 
student.  The  squares  within  circles  are  student  events,  such  as  Student  Problem  Solving 
Action,  the  category  that  captured  the  actions  while  students  solved  problems  and  tried  to 
interpret  the  text.  The  frequency  of  each  event  is  listed  in  Appendix  B. 

The  most  important  source  of  information  in  Figure  1  is  the  arrows.  These  arrows 
show  that  some  event  type  followed  some  other  event  type.  To  construct  the  figure,  we 
examined  the  frequency  with  which  each  event  type  followed  all  other  types  of  events.  We 
then  examined  this  transition  analysis  (Bakeman  k  Gottman.  1986;  Fisher.  1991)  for  the 
most  common  chronological  sequences,  enabling  us  to  specify  precisely  what  sorts  of  events 
occurred  and  their  relationships  during  the  sessions.  The  arrows  represent  all  the  transitions 
between  one  event  and  another  that  occurred  more  often  than  10  times  in  the  protocols. 
The  wide  arrows  are  the  most  frequent  transitions  in  the  data,  those  that  occurred  more 
than  100  times.  For  example.  Tutor  Confirm  Step  often  followed  Student  Problem  Solving 
Actions.  For  completeness,  the  entire  transition  matrix  is  shown  in  Appendix  C. 

In  addition  to  the  combined  categories  SPSA  and  TCS.  Figure  1  contains  a  category 
called  Tutor  Error  Feedback.  As  described  earlier,  the  manners  in  which  tutors  respond  to 
errors  can  be  crucial  for  learning.  For  the  initial  picture  of  the  tutoring  sessions  shown  in 
Figure  1.  we  merged  three  categories  of  explicit  error  feedback,  Tutor  Correction,  Tutor 
Surface  Feature  Feedback,  and  Tutor  Plan-Based  Feedback,  into  Tutor  Error  Feedback.  The 
categories  comprising  Tutor  Error  Feedback  contain  only  tutorial  utterances  that  provide 
direct  and  explicit  guidance  after  errors,  but  these  events  do  not  exhaust  all  possible  tutorial 
responses  to  student  errors.  Any  utterance  could  in  principle  be  used  in  response  to  an 
error.  We  will  address  how  tutors  respond  to  errors  in  the  next  analysis. 
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Since  we  believe  that  most  of  learning  occurs  during  the  sort  of  problem  solving  actions 
captured  in  Student  Problem  Solving  Action  (Anderson.  1983:  Laird  et  ah.  1986:  VanLehn. 
1988).  we  will  focus  on  Student  Problem  Solving  Action  for  the  remainder  of  this  paper. 
Specifically,  we  shall  next  present  two  sorts  of  analyses  of  tutorial  assistance.  First,  we  will 
discuss  the  means  by  which  tutors  help  keep  the  students  problem  solving  on  productive 
paths  via  confirmatory  feedback,  error  feedback,  and  other  guidance.  We  then  turn  to  a 
finer  examination  of  errors  to  uncover  the  ways  tutors  offer  feedback,  to  see  if  tutors  in  fact 
respond  differently  depending  upon  the  nature  of  the  student's  error,  and  to  examine  how 
the  student  and  tutor  work  together  to  repair  errors. 

How  tutors  keep  problem  solving  productive 

In  this  section,  we  examine  the  strategies  tutors  use  to  provide  guidance  to  students 
while  they  are  in  the  process  of  understanding  and  solving  problems,  that  is.  engaged  in 
Student  Problem  Solving  Actions.  This  section  focuses  on  the  ways  tutors  help  keep  student 
problem  solving  productive  and  continuing.  We  look  initially  at  correct  problem  solving 
actions  to  see  how  tutors  respond  and  then  turn  to  responses  following  impasses  and  errors. 

Confirmatory  Feedback 

How  do  tutors  respond  when  students  make  correct  problem  solving  actions?  Fox  (1991) 
argued  that  tutors  offer  confirmatory  feedback  after  correct  steps.  Our  category  Tutor 
Confirm  Step  captured  this  sort  of  tutorial  feedback.  There  were  3.506  Student  Problem 
Solving  Actions  in  our  data.  Of  these.  1.495  (44%)  were  followed  immediately  by  a  Tutor 
Confirm  Step.  This  information  is  represented  by  the  wide  arrow  from  Student  Problem 
Solving  Action  to  Tutor  Confirm  Step  in  Figure  1.  The  remaining  56%  of  transitions 
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from  Student  Problem  Solving  Action  were  primarily  made  up  of  occurrences  of  one  SPSA 
following  another  and  Tutor  Guidance  utterances  following  an  SPSA. 

Notice  that  the  44%  was  calculated  using  all  Student  Problem  Solving  Actions,  includ¬ 
ing  erroneous  steps.  When  considering  only  the  2.261  correct  problem  solving  actions,  the 
picture  becomes  even  more  striking  —  66%  of  correct  Student  Problem  Solving  Actions 
received  confirmatory  feedback.  Almost  no  incorrect  steps  received  confirmatory  feedback. 
This  indicates  that  tutors  are  commonly  confirming  correct  actions,  but  carefully  not  of¬ 
fering  confirmatory  feedback  after  erroneous  actions. 

These  problem  solving  steps  are  not  by  and  large  complete  solutions.  This  high  propor¬ 
tion  of  confirmations  does  not  reflect  situations  in  which  the  student  has  created  an  entire 
solution  to  which  the  tutor  responds.  “Yes.  Most  solutions  required  multiple  problem  solv¬ 
ing  actions,  even  when  no  errors  occurred.  In  fact,  problems  that  require  definitions  of  new 
LISP  functions,  as  in  the  second  and  third  chapters,  required  an  average  of  10  correct 
events  to  complete,  even  with  no  erroneous  events.  These  problems  received  an  average 
of  six  Tutor  Confirm  Steps  per  problem  as  well.  Thus,  tutors  offered  confirmations  verv 
often  —  66%  of  correct  events  —  and  these  confirmations  occurred  during  ongoing  problem 
solving,  rather  than  at  its  successful  completion. 

To  further  emphasize  that  students  received  these  confirmations  during  problem  solv¬ 
ing,  note  that  43%  of  Tutor  Confirm  Steps  were  followed  immediately  by  another  Student 
Problem  Solving  Action.  Table  6  gives  an  example  of  the  use  of  Tutor  Confirm  Steps  during 
attempts  to  solve  a  problem  called  first- eiem,  The  student  initially  set  up  the  first  part  of 
the  expression  to  be  typed,  and  the  tutor  responded  with  a  confirmation  (TCS).  The  tutor 
offers  further  confirmations  as  problem  solving  continues  through  the  problem. 
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Insert  Table  6  about  here 

It  might  be  suggested  that  these  confirmations  are  in  fact  simply  conversational  require¬ 
ments.  since  people  are  expfected  to  respond  to  others*  statements  (Grice.  1975).  If  this 
were  the  case,  tutors  would  have  confirmed  all  steps.  However,  tutors  did  not  confirm  all 
steps,  but  in  fact  only  offered  confirmations  to  correct  steps  and  responded  to  errors  with 
other  types  of  tutorial  actions.  Thus,  tutor  confirmatory  feedback  is  indeed  informative, 
and  tells  the  student  that  the  previous  action  fell  onto  a  profitable  solution  path. 

Tutors  could  also  encourage  students  to  continue  on  the  current,  productive  solution 
path  via  Tutor  Guidance.  A  tutor  response  such  as.  “So  next  we  need  the  else  case 
includes  an  implicit  confirmation  of  the  previous  step.  In  other  words,  the  tutor  is  basically 
saying  uOK  so  far,  now  the  else  case  is  next/’  Tutor  Guidance  could  include  utterances  that 
are  motivational  in  nature,  such  as  “Yeah,  you're  doing  just  fine,  which  also  encourage 
the  student  to  continue  along  the  correct  path.  Tutor  Guidance  events  make  up  another 
16%  of  events  subsequent  to  the  3.506  SPSAs.  and  make  up  an  additional  70%  of  events 
following  the  correct  SPSAs.  Thus.  Tutor  Guidance  utterances  that  contain  an  implicit 
confirmation  are  another  important  manner  in  which  tutors  keep  problem  solving  productive 
by  encouraging  students  to  continue  on  a  productive  solution  path. 

These  results  support  the  claims  of  Fox  (1991).  They  show  that  tutors  do  in  fact  offer 
confirmatory  feedback  throughout  the  problem  solving  process.  Correct  events  usually  re¬ 
ceive  confirmations  immediately,  and  incorrect  events  do  not  receive  confirmatory  feedback. 
This  type  of  encouragement  when  students  are  on  a  promising  solution  path  appears  to  be 
one  method  by  which  tutors  help  guide  students*  problem  solving.  We  next  turn  to  the 
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tutorial  responses  that  followed  errors. 

Responses  to  incorrect  problem  solving  steps 

Not  all  solution  steps  are  correct.  Errors  offer  a  particularly  crucial  opportunity  for 

i 

learning.  Some  researchers  have  argued  that  recovering  from  errors  carries  great  potential 
dangers  (Anderson.  1983;  Lewis  Sc  Anderson.  1985:  Sweller.  1988).  while  others  have  argued 
that  recovering  from  and  explaining  impasses  is  the  key  to  effective  learning  (Chi  et  al., 
1989:  Laird  et  ah.  1986;  Schank  Sc  Leake.  1989:  YanLehn.  1990).  First  we  consider  how 
errors  are  uncovered  in  the  problem  solving  sessions.  How  do  tutors  help  students  recover 
from  errors?  Do  tutors  allow  students  to  locate  and  repair  their  own  errors,  a  strategy 
emphasized  by  some  learning  researchers  (Papert,  1980;  Schank  Sc  Leake,  1989),  or  do 
tutors  find  the  errors  for  the  students? 

Errors  were  not  left  unnoticed  for  very  long.  In  fact.  75%  of  all  errors  in  the  sessions 
were  indicated  within  two  events.  Typically,  an  error  occurred,  the  student  or  tutor  made 
one  other  utterance,  and  then  the  error  was  flagged.  Recall  that  an  error  flag  is  the  initial 
indication  in  the  discourse  that  an  error  has  occurred,  and  the  flag  could  be  any  of  the 
different  types  of  events  we  coded,  such  as  a  Tutor  Focus  Attention:  ’‘Umm,  look  back  up 
there  —  there  might  be  a  problem. Thus,  these  problem  solving  sessions  are  not  typified  by 
long  exploratory  searches,  during  which  errors  may  occur  and  not  be  noticed  immediately, 
a  pedagogical  approach  sometimes  advocated  fe.g..  Papert.  1980).  Table  7  shows  a  typical 
example  of  a  student  flagging  her  own  error,  and  Table  8  shows  a  tutor  flagging  a  student 
error. 

Insert  Table  7  about  here 
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Insert  Table  8  about  here 

Since  error  repair  was  begun  very  rapidly,  we  next  turn  to  the  question  of  who  noticed 
the  errors.  Fox  (1991)  and  I/epper  et  al.  (1990)  argued  that  the  student  plays  a  major  role 
in  locating  and  repairing  errors.  In  fact,  of  the  1.224  errors  identified  for  analysis,  almost 
half  (47%)  were  noticed  by  the  student.  What  sorts  of  errors  did  the  students  catch? 
Interestingly,  most  of  the  errors  caught  by  the  students  (91%)  were  typographical  errors  or 
syntactically  incorrect  LISP  expressions.  Although  there  may  be  pedagogical  advantages 
for  students  to  find  and  repair  many  different  sorts  of  errors,  including  errors  involving 
problem  goals,  for  example,  students  generally  did  not  do  so.  Either  they  simply  could  not 
find  these  errors  or  our  tutors  did  not  allow  them  the  leeway  to  do  so.  Next  we  consider 
how  tutors  responded  to  student  errors. 

Tutors  flagged  approximately  53%  of  all  errors.  Of  the  errors  flagged  by  the  tutor,  only 
41%  were  typographical  or  syntactic  errors.  The  remaining  59%  were  errors  relating  to 
goals  and  to  the  meanings  of  LISP  operators.  Thus,  tutors  mainly  caught  the  more  difficult 
errors,  whereas  students  mainly  caught  the  low-level  errors. 

Insert  Figure  2  about  here 

Figure  2  shows  the  number  of  errors  flagged  by  the  student  and  the  tutor  as  a  function 
of  the  number  of  events  intervening  between  the  error  and  the  repair.  Most  student-flagged 
errors  were  caught  very  quickly  —  often  on  the  same  event,  with  a  lag  of  zero  while 
many  of  the  tutor-flagged  events  were  more  removed  from  the  impasse.  Most,  but  not  all. 
of  the  tutor  flags  occurred  very  quickly  after  the  error,  usually  on  the  next  event.  Thus. 
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tutors  flagged  most  of  the  more  serious  errors  related  to  goals  and  to  the  meanings  of  LISP 
operators,  and  generally  did  not  allow  students  to  find  and  repair  their  own  errors,  since 
the  tutors  commented  on  most  errors  immediately  after  the  error  occurred  if  the  student 
did  not  notice  it. 

Analysis  of  the  events  following  the  error  flags  suggest  that  the  tutors  error  flags  were 
used  during  the  students'  ongoing  problem  solving.  Sixty-three  percent  of  tutor  error  flags 
were  followed  immediately  by  another  Student  Problem  Solving  Action,  thereby  indicating 
that  students  received  the  assistance  and  began  implementing  a  repair.  Almost  all  the 
remaining  tutor  error  flags  were  followed  by  Student  Confirmations.  Although  some  have 
suggested  that  students  have  great  difficulty  understanding  error  feedback  given  by  instruc¬ 
tors  (Graesser  et  ah,  1993:  Moore  Sz  Swartout.  1989),  the  students  in  this  study  seemed  to 
understand  the  error  flags  quite  well,  since  most  tutor  error  flags  were  not  followed  by  re¬ 
quests  for  clarification  and  elaboration.  Thus,  error  flags,  like  confirmations,  are  important 
to  the  ongoing  problem  solving  process. 

These  analyses  have  suggested  that  tutors  did  not  allow  students  a  great  deal  of  time  to 
discover  their  errors  -  if  the  student  did  not  comment  on  an  error  essentially  immediately 
after  it  happened,  the  tutor  did.  Despite  the  potential  benefits  of  students  finding  and 
repairing  their  own  errors,  students  actually  found  mostly  syntactic  low-level  errors.  Repair 
of  the  more  complex  errors  was  at  least  initiated  by  the  tutors.  In  subsequent  analyses  in 
the  next  section,  we  consider  the  type  of  guidance  tutors  provided  on  errors  and  how  the 
error  recovery  episode  progressed. 


34 


Tutoring:  Guided  learning  by  doing 


Summary  of  tutorial  guidance 

In  this  section,  we  considered  how  tutors  offer  interactive  support  for  student  problem 
solving.  We  showed  that  tutors  offer  rapid  confirmations  and  supportive  guidance  to  cor¬ 
rect  student  actions,  even  sinall  actions  like  interpreting  a  short  text  section  or  creating 
individual  components  of  a  solution.  These  confirmations  are  more  than  the  conversational 
politeness  of  acknowledging  the  other  speaker,  since  the  tutors  offered  confirmations  onl\ 
to  correct  steps.  In  contrast,  incorrect  steps  were  flagged  very  rapidly.  Students  caught 
almost  half  of  the  errors,  but  the  errors  they  noticed  were  primarily  low  level  errors  such 
as  typing  mistakes  or  syntactic  problems.  The  remainder  of  the  errors  were  caught  by  the 
tutor,  usually  within  one  or  two  events.  Thus,  tutors  are  very  much  involved  in  the  ongo¬ 
ing  problem  solving,  since  most  student  steps  received  some  sort  of  tutorial  response  that 
helped  guide  problem  solving,  either  a  confirmation  or  notification  of  an  error. 

These  analyses  support  a  view  of  tutoring  as  guided  learning  by  doing.  When  a  student 
solves  problems  alone,  it  is  usually  difficult  to  determine  whether  a  step  was  correct  or 
not,  and  the  student  may  not  be  sure  it  was.  However,  when  a  tutored  student  makes 
a  correct  step,  the  tutor  intervenes  to  say  it  was  correct,  thus  helping  problem  solving 
continue.  Also,  when  a  student  working  alone  makes  an  error,  it  may  not  be  noticed  for 
some  time,  making  repair  difficult  and  floundering  likely.  Tutors  ensured  that  any  errors 
were  noticed  very  quickly,  jumping  in  to  tell  the  student  the  step  was  incorrect,  if  needed. 
These  confirmations  and  error  flags  serve  to  guide  the  ongoing  problem  solving  and  keep  it 
productive. 

Having  focused  on  the  ways  tutors  help  keep  ongoing  problem  solving  producti\e.  we 
next  turn  to  a  more  precise  examination  of  the  errors  made  during  the  learning  sessions. 
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focusing  on  the  type  of  assistance  tutors  provided  when  they  offered  feedback.  We  will 
describe  the  ways  tutors  and  students  worked  together  to  repair  errors,  paying  particular 
attention  to  the  ways  the  tutors  and  students  dealt  with  different  types  of  errors. 

Errors  and  the  content  of  feedback 

There  has  been  a  great  deal  of  focus  on  the  manner  in  which  tutors  scaffold  students’ 
recovery  from  errors  and  impasses.  For  example.  Fox  (1991)  and  Lepper  et  ah  (1990) 
argued  that  tutors  attempt  to  indicate  errors  to  the  student  subtly,  so  that  the  student  can 
perform  the  repair,  while  Littman  et  al.  (1990)  argued  that  tutors  offer  quite  explicit  error 
feedback.  The  content  and  style  of  error  feedback  can  have  significant  effects  on  learning 
and  motivation  (Lepper  et  al.,  1990;  McKendree,  1990;  Reiser,  Copen,  Ranney,  Hamid,  &; 
Kimberg,  1994),  and  thus,  understanding  how  tutors  respond  to  student  errors  can  cast 

further  light  on  how  tutors  guide  their  students. 

Recovering  from  an  impasse  or  error  entails  several  components  (Merrill  et  al.,  1992). 
The  first  stage  of  recovering  from  an  impasse  is  realizing  that  an  error  has  occurred.  Then, 
the  erroneous  features  of  the  solution  must  be  located,  and  the  erroneous  portion  must  be 
replaced  with  a  successful  fulfillment  of  the  goal.  Finally,  it  may  even  be  helpful,  though 
not  necessary,  to  understand  why  the  error  occurred.  These  components  are  fundamentally 
distinct,  even  though  they  usually  occur  as  a  group.  A  student  might  perform  all  of  the 
components,  thereby  completing  all  of  the  error  recovery.  At  the  other  extreme,  a  tutor 
might  tell  a  student  exactly  what  went  wrong  and  how  to  repair  it.  The  error  recover} 
process  could  also  be  a  collaborative  enterprise,  with  the  tutor  scaffolding  the  recovery  as 
needed  to  get  problem  solving  back  on  track,  but  allowing  the  student  to  perform  much 
of  the  work.  If  the  tutors  main  goal  is  to  get  the  problem  solving  back  on  track,  we 
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would  expect  feedback  to  perform  most  of  the  error  recovery  components  for  the  students. 
Alternatively,  if  errors  are  used  as  opportunities  for  learning,  tutors  might  allow  students 
to  perform  most,  if  not  all.  of  this  process. 

In  this  section,  we  inveslpgate  whether  there  are  any  regularities  in  tutorial  responses 
to  errors.  We  first  define  the  types  of  errors  students  committed,  and  then  present  the 
different  types  of  error  feedback  tutors  used  to  initiate  error  recovery  in  our  categorization 
scheme.  Next,  we  examine  the  relationship  between  tutor  error  initiations  and  error  type. 
In  the  final  part  of  this  section,  we  discuss  how  students  and  tutors  collaborated  to  repair 
errors. 

Before  describing  tutorial  responses  to  errors,  we  first  describe  the  types  of  errors  in 
detail.  Recall  that  two  independent  coders  identified  all  errors  in  the  verbalizations  and 
typing  (see  Table  5).  Some  of  these  errors  represent  difficulties  in  planning  a  solution,  others 
involve  incorrect  assertions  about  functions  and  concepts  in  LISP,  and  still  others  reflect 
difficulties  in  implementing  a  correct  solution.  These  three  categories  of  errors  capture 
a  continuum  of  behavior  ranging  from  planning  difficulties  to  problems  implementing  a 
solution.  This  will  allow  us  to  see  whether  tutors  respond  differently  to  errors  where  the 
repairs  have  differing  severity.  The  present  analysis  excludes  the  360  typographical  errors 
which  were  typically  slips  and  self-corrected  by  the  student.  We  focus  on  the  remaining 
categories  of  errors.  These  five  categories  form  three  groups  that  we  will  use  as  the  basis 
for  this  analysis:  syntactic  errors,  semantic  errors,  and  goal  errors. 

The  first  category,  syntactic  errors  are  difficulties  in  communicating  an  expression  to  the 
LISP  interpreter  correctly  so  that  the  expression  can  be  understood.  This  category  covers 
cases  where  the  programming  constructs  and  algorithms  used  would  achieve  the  stated 
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goal,  but  the  student  implemented  a  construct  incorrectly  in  the  language.  These  errors 
consisted  only  of  the  addition  of  extra  parentheses,  the  deletion  of  needed  parentheses,  or 

the  addition  or  deletion  of  quotation  marks. 

The  second  class  of  errors,  semantic  errors,  are  inappropriate  uses  of  LISP  constructs, 
and  includes  error  types  Semantic:  Operator  and  Semantic:  Concept  shown  in  Table  5. 
These  covered  expressions  that  were  syntactically  correct,  but  made  use  of  inappropriate 
constructs.  One  way  of  misapplying  a  construct  is  to  apply  a  function  when  the  precondi¬ 
tions  of  the  function  are  not  met.  An  example  of  a  precondition  failure  occurred  when  a 
student  typed  (cons  'a  ’b)  to  construct  a  list:  in  this  curriculum,  cons  requires  a  list  as  its 
second  argument,  but  the  student  used  an  atom.  To  consider  another  example,  the  student 
might  claim  incorrect  output  of  a  function  that  could  be  applied  to  the  data.  For  example, 
one  student  said  “(: member  ‘ b  ’(a  b  c  d))  returns  (b)n ,  when  in  fact  it  would  return  the  list 
(bed).  Thus,  although  member  can  be  applied  in  this  situation,  the  student  did  not  applj 
it  correctly.  These  semantic  errors  consist  of  more  than  simply  a  failure  in  communicating 
a  solution  to  the  computer,  they  are  erroneous  choices  or  applications  of  that  which  must 

be  communicated  —  operators  in  the  domain. 

The  third  category  of  errors.  Goal  Errors,  consists  of  errors  concerning  setting  or  ful¬ 
filling  goals.  These  errors  included  categories  Goal:  Incorrect  and  Goal:  Skipped  Goal  in 
Table  5.  In  these  cases,  a  solution  was  syntactically  correct  and  used  the  correct  function 
for  a  goal,  but  the  goal  under  consideration  was  in  some  way  incorrect.  When  reading 
the  problem,  the  student  must  try  to  set  up  an  initial  goal  structure  to  begin  solving  the 
problem.  The  student  might  have  difficulty  or  make  mistakes  doing  this,  as  reflected  by 
the  student  who  said  “. . .  so  I  just  return  the  [first!  variable.’  when  the  correct  subgoal  was 
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to  return  a  list  of  the  first  and  second  variables.  In  other  cases,  students  were  solving  the 
problem,  choosing  and  achieving  subgoals,  and  failed  to  implement  a  subgoal  that  had  been 
previously  set.  In  these  cases,  students  were  able  to  set  up  a  goal  structure,  but  then  failed 
to  achieve  a  goal  that  had  bpen  present  in  the  structure  at  the  start.  These  errors  concern 
an  even  more  critical  component  of  problem  solving  than  Semantic  Errors.  They  involve 

planning  a  solution  apart  from  its  implementation. 

Having  presented  the  three  classes  of  student  errors,  we  can  now  consider  how  tutors 
responded  to  each  type.  First,  we  consider  the  events  used  to  initiate  error  recovery  Fol¬ 
lowing  that  analysis,  we  investigate  how  the  type  of  feedback  was  related  to  the  type  of 
error.  Recall  that  Figure  1  included  a  category  called  Tutor  Error  Feedback.  This  category 
is  made  up  of  three  categories,  Tutor  Correction.  Tutor  Surface  Feature  Feedback,  and  Tu¬ 
tor  Plan  Based  Feedback.  These  differ  in  the  portions  of  the  error  recovery  process  initially 
performed  by  the  tutor.  When  tutors  intervene,  they  must  determine  how  much  of  the  error 
recovery  process  they  will  perform.  Analyzing  the  occurrence  of  tutorial  error  feedback  will 
cast  light  on  how  tutorial  feedback  is  tailored  to  student  errors.  First  we  review  the  three 
categories  of  tutorial  feedback. 

We  defined  error  feedback  as  utterances  that  contained  a  reference  to  incorrect  features 
of  the  student’s  solution.  The  feedback  might  also  contain  information  about  how  to  repair 
the  error.  Although  there  was  usually  only  one  error  feedback  per  error,  our  coding  scheme 
allowed  for  multiple  feedback  per  error.  Thus,  our  definition  of  error  feedback  is  very  similar 
to  our  definition  of  error  flag,  except  that  a  flag  might  not  point  to  any  particular  feature  of 
the  solution,  while  an  instance  of  error  feedback  must  point  to  a  feature.  Tutors  used  one 
of  the  error  feedback  categories  to  flag  errors  in  95%  of  cases.  However,  because  here 
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are  concerned  with  the  information  conveyed  by  the  tutor  in  response  to  a  student  error,  in 
this  analysis  we  examine  the  tutorial  error  feedback  instead  of  error  flags.  As  will  be  seen 
below,  tutors  could  indicate  features  at  differing  levels  of  abstraction  and  with  differing 
amounts  of  information  aboyt  the  repair. 

We  categorized  feedback  as  Tutor  Correction  if  the  tutor  responded  to  a  student  error 
by  telling  the  student  what  the  correct  action  should  have  been  and  how  to  repair  the 
error.  Thus,  this  category  captures  tutorial  utterances  that  perform  all  of  the  components 
of  the  error  recovery.  A  Tutor  Correction  might  contain  an  explanation  of  why  the  step  was 
incorrect,  or  it  might  simply  be  a  directive.  For  example,  one  tutor  said  You  want  to  quote 
that,  since  it’ll  be  a  function  call  otherwise 5  after  the  student  typed  flistp  (a  b  c  d )).  This 
tells  the  student  that  a  quotation  mark  before  the  (abed)  has  been  forgotten,  and  why 
that  matters.  In  a  different  situation,  a  tutor  said  “You’ll  need  to  use  and  there,  instead 
of  or.”  The  utterances  differ  with  respect  to  how  much  explanation  is  given  along  with 
the  correction,  but  both  are  Tutor  Corrections,  since  each  tells  the  student  exactly  how  to 
repair  the  impasse. 

We  also  had  two  categories  for  error  feedback  that  initially  provided  fewer  of  the  error 
recovery  components.  An  utterance  was  considered  Tutor  Surface  Feature  Feedback  if  it 
only  pointed  out  an  erroneous  feature  explicitly  present  in  the  student  s  solution.  For 
example,  instead  of  offering  a  Tutor  Correction  to  the  listp  example  above,  the  tutor  could 
have  offered  a  Tutor  Surface  Feature  Feedback  by  saying  “An  unquoted  list  is  a  function 
call.”  This  type  of  feedback  points  out  to  the  student  where  the  problem  is,  and  often 
includes  information  about  which  component  is  incorrect,  but  does  not  directly  suggest  a 
repair.  The  repair  may  be  easily  inferable  from  the  feedback,  as  in  the  last  example  in 
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which  the  repair  is  to  add  a  quotation  mark,  but  an  inference  is  required,  nonetheless.  For 
example,  the  students  read  in  the  textbook  that  cons  took  two  arguments,  an  atom  as  the 
first  argument  and  a  list  as  the  second.  When  a  student  began  typing  an  expression  designed 
to  rotate  the  last  element  of  a  list  to  the  front  using  the  function  cons ,  and  typed  (cons 
(last  lis ).  the  tutor  intervened  to  say  '"Last  returns  a  list,  not  an  atom.  The  student  had  to 
infer  what  the  tutor  meant  by  the  feedback.  This  feedback  could  indicate  that  either  last 
or  cons  was  the  wrong  function  to  use,  or  that  the  student  had  forgotten  some  additional 
function  that  needed  to  be  used,  or  even  that  the  last  should  have  been  the  second  argument 
to  cons  instead  of  the  first.  All  of  these  inferences  are  the  potential  intent  of  the  tutorial 
feedback.  The  feedback  itself  serves  to  make  the  student  aware  of  the  general  location  of  an 
error.  The  student  must  infer  the  nature  of  the  error  from  the  feedback,  set  a  goal  for  the 
repair,  and  begin  replacing  the  errant  portion  of  the  solution.  Thus,  students  have  more 
opportunity  to  participate  in  the  error  repair  after  a  Tutor  Surface  Feature  Feedback  than 
after  a  Tutor  Correction. 

Finally,  the  tutor  may  leave  even  more  of  the  error  repair  to  the  student.  The  category 
Tutor  Plan-Based  Feedback  simply  restated  the  goal  the  student  should  have  been  pursu¬ 
ing.  For  example,  a  tutor  said  “The  function  should  return  found  if  the  item  is  in  the  list 
after  a  student  coded  a  function  that  returned  t  in  that  case.  This  tells  the  student  that, 
although  the  programming  constructs  employed  may  be  used  appropriately,  a  goal  embod¬ 
ied  by  that  portion  of  the  program  is  incorrect.  Once  again,  even  though  the  inference 
required  to  determine  which  is  the  incorrect  goal  may  be  fairly  straightforward,  the  student 
is  still  required  to  make  an  inference.  Furthermore,  an  utterance  such  as  You  want  to 
see  if  both  arguments  are  lists,  not  if  either  one  is  a  list'1  in  response  to  the  student  code 
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(or  for  (listp  orgl)  (listp  arg2)))  is  not  as  clear.  In  fact,  this  student  needed  to  replace  the 
second  or  with  an  and.  However,  the  student  might  sensibly  infer  that  the  correct  action 
was  to  have  onlv  one  or,  for  example,  and  therefore  erroneously  to  delete  the  first  or.  This 
type  of  feedback  requires  th^  student  first  to  localize  the  difference  between  the  goal  the 
tutor  savs  and  the  goal  the  student  was  pursuing,  and  then  replace  the  erroneous  portion  of 
the  solution.  Thus,  the  student  has  even  more  opportunity  in  this  situation  to  participate 
in  the  error  recovery  process  than  in  the  two  previous  feedback  types. 

Insert  Figure  3  about  here 

For  this  analysis  we  considered  only  those  575  errors  (95%  of  the  tutor-caught  errors) 
that  received  one  of  these  types  of  explicit  tutorial  error  feedback,  excluding  the  42  typo¬ 
graphical  errors  caught  by  the  tutor  and  the  33  errors  that  did  not  receive  one  of  the  three 
forms  of  explicit  error  feedback.  We  then  examined  the  frequency  that  each  type  of  student 
error  elicited  from  each  of  the  three  types  of  tutorial  response.  This  analysis,  shown  in 
Figure  3,  reveals  a  strong  relationship  between  the  nature  of  the  student  s  error  and  the 
type  of  feedback  provided  by  the  tutor,  yy  >  150,  p  <  .001.  The  tutors  exhibited  a  strong 
tendency  to  intervene  with  a  different  guidance  strategy  depending  upon  the  nature  of  the 
error. 

When  the  error  consisted  only  of  a  low-level  syntactic  detail,  the  tutors  prevented  the 
students  from  floundering  by  suggesting  exactly  how  to  repair  the  error.  Table  9  presents  a 
typical  example  of  this  type  of  student  error  and  tutorial  response  episode.  In  this  example, 
line  2.  the  tutor  told  the  student  to  insert  a  quotation  mark  the  student  had  omitted. 

Insert  Table  9  about  here 
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In  contrast  to  syntactic  errors,  most  semantic  errors  received  Surface  Feature  Feedback. 
In  these  cases,  the  tutor  pointed  out  the  erroneous  feature  of  the  solution  to  the  student 
rather  than  suggesting  an  explicit  repair.  Table  10  presents  a  typical  example  of  a  tutor 
focusing  on  a  feature  of  th^  student's  solution  in  this  manner.  Here,  in  line  6,  the  tu¬ 
tor  described  the  correct  behavior  of  append,  indicating  it  should  not  be  applied  in  this 
situation. 

Insert  Table  10  about  here 

The  third  type  of  student  error,  goal  errors,  also  received  Tutor  Corrections  and  Tutor 
Surface  Feature  Feedback.  But  this  type  of  error  also  received  a  substantial  amount  of  Plan- 
Based  Feedback,  which  did  not  often  occur  in  response  to  the  other  error  types.  Table  11 
presents  a  typical  example  in  which  the  tutor  comments  on  the  students  goals.  Here,  the 
student  used  an  incorrect  order  of  conditional  cases  in  the  solution,  and  the  tutor  reminded 
the  student  of  the  importance  of  case  order  (on  line  2). 

Insert  Table  11  about  here 

These  results  suggest  an  interesting  relationship  between  the  type  of  error  and  the  tu¬ 
tor’s  initial  response  to  the  error.  In  those  cases  when  the  error  occurred  while  trying  to 
communicate  a  solution  to  the  computer,  the  tutors  did  virtually  all  of  the  error  recovery 
process.  However,  if  the  error  concerned  the  selection  of  an  operator  in  the  domain,  the  tu¬ 
tors  pointed  out  the  erroneous  feature  to  the  student  and  allowed  the  student  to  participate 
in  the  remainder  of  the  error  recovery.  If  the  error  was  at  an  even  higher  level,  constructing 
and  maintaining  the  problem’s  goal  structure,  the  tutor  assisted  with  the  subgoal  selection 
by  offering  a  Tutor  Plan-Based  Feedback,  thus  providing  the  student  the  opportunity  to 
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identify  the  errors,  plan  the  recovery,  and  execute  it.  In  these  cases,  the  tutor  performed 
little  of  the  recovery  process,  allowing  the  student  to  do  most  of  the  recovery. 

These  results  suggest  very  clearly  that  the  effectiveness  of  tutorial  feedback  may  arise 
because  of  the  contingency  o£ feedback  style  and  content  on  the  nature  of  the  student  s  error. 
However,  one  alternative  possibility  to  be  considered  is  whether  the  categories  of  tutorial 
response  were  defined  so  that  each  was  logically  possible  only  on  a  subset  of  error  types. 
For  example,  perhaps  tutors  could  only  offer  Tutor  Corrections  in  response  to  syntactic 
errors. 

In  fact,  however,  tutors  did  offer  all  types  of  feedback  to  all  types  of  errors,  albeit 
with  differing  frequencies.  To  further  illustrate  this  point.  Table  12  contains  examples  of 
tutorial  feedback  demonstrating  a  plausible  tutorial  response  of  each  type  to  each  type 
of  student  error.  These  examples  are  taken  directly  from  our  protocols,  slightly  modified 
so  that  each  example  refers  to  the  same  error,  making  it  easier  to  compare  the  different 
feedback  examples.  The  original  feedback  was  of  the  type  presented  in  the  table  (e.g.,  Plan- 
Based  Feedback),  and  did  refer  to  the  same  type  of  error  (e.g.,  syntactic  error),  however. 
Notice  that  tutors  were  able  to  offer  Plan-Based  Feedback,  even  to  a  syntactic  error  (1c). 
This  feedback  refers  to  the  goal  the  student  should  have  been  working  on,  thus  allowing 
the  student  to  identify  and  correct  the  error,  but  refers  to  the  syntactic  error  of  adding 
an  undesired  quotation  mark  before  the  variable  Its .  Thus,  our  three  tutorial  feedback 
categories  are  not  biased  in  their  definitions  to  restrict  the  feedback  to  particular  error 
types.  Instead,  the  pattern  in  the  data  appears  to  represent  a  real  aspect  of  individualized 
instruction.  In  the  next  section,  we  detail  the  student  involvement  in  the  error  recover} 
process,  and  highlight  the  differing  outcomes  of  each  type  of  tutorial  feedback. 
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Insert  Table  12  about  here 

What  did  students  do  in  the  error  recovery? 

We  have  focused  so  far  on  the  role  of  tutorial  guidance,  but  we  have  not  yet  considered 
the  role  that  students  play  in  the  error  repair.  One  possible  scenario,  given  the  active 
nature  of  the  tutor’s  guidance,  is-  that  students  play  an  active  role  when  solving  problems 
but  switch  into  a  subsidiary  role  when  errors  occur,  following  the  tutor  s  directions,  perhaps 
asking  questions  to  clarify  advice  (cf..  Moore  Swartout.  1989),  but  not  playing  an  active 
role  in  the  recovery.  In  this  section,  we  examine  the  role  of  students  in  the  error  recovery 
process. 

Insert  Figure  4  about  here 

In  our  analysis  of  tutorial  feedback,  we  suggested  that  feedback  varied  in  the  portion  of 
the  error  recovery  process  left  to  perform  after  the  feedback.  To  confirm  this,  we  analyzed 
the  median  number  of  events  required  to  repair  an  error  after  it  occurred.  If  there  more  of 
the  error  recovery  process  remains  after  a  Tutor  Plan-Based  Feedback  than  after  a  Tutor 
Correction,  more  events  should  be  required  to  achieve  the  subgoal  correctly  after  a  Tutor 
Plan-Based  Feedback.  Indeed,  as  shown  in  Figure  4,  it  took  more  events  to  repair  a  goal 
error,  requiring  a  median  of  four  additional  events,  than  a  semantic  error,  which  required 
three  events.  Syntactic  errors  exhibited  the  shortest  repairs,  requiring  a  median  of  two 
events  after  the  error  occurred.  These  longer  durations  of  repair  episodes  suggest  increased 
difficulty  forming  a  repair,  consistent  with  our  finding  that  more  of  the  error  recovery 
process  was  left  to  perform. 
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Analysis  of  the  events  that  occurred  during  these  error  episodes  reveals  a  collaborative 
repair  process.  Although  errors  were  typically  repaired  very  quickly,  sometimes  after  as 
little  as  one  event,  the  repair  process  did  not  typically  consist  of  the  tutor  leading  a  passive 
student  back  on  to  a  productive  solution  path.  Instead,  students  attempting  to  repair  an 
error  proposed  actions  to  take,  and  then  the  tutor  and  student  worked  together  to  get  the 
solution  back  on  track. 

Insert  Figure  5  about  here 

Figure  5  displays  the  common  sequences  of  events  following  an  error,  those  occurring 
more  than  50  times  in  all  protocols.  The  student  error  is  in  the  upper  left  corner  of  the 
figure,  and  the  next  event  wa s  usually  one  of  the  types  of  tutor  error  feedback.  Infrequenth. 
there  were  a  few  student  actions  intervening  between  the  error  and  the  feedback.  These 
are  represented  by  the  Other  Student  Problem  Solving  Actions  box  in  the  upper  right 
hand  corner,  with  the  dotted  links  representing  infrequent  events.  The  collaborative  repair 
typically  began  with  the  Tutor  Error  Feedback.  A  Student  Problem  Solving  Action  usually 
followed  the  feedback,  presumably  implementing  a  partial  repair.  The  tutor  often  gave 
confirmatory  feedback  to  the  SPSA,  to  which  the  student  offered  a  confirmation  also,  and 
the  tutor  offered  guidance,  perhaps  in  the  form  of  a  new  goal  to  be  achieved,  that  the  student 
attempted  to  implement  with  another  SPSA.  This  is  a  clear  collaborative  enterprise,  with 
the  student  actively  working  to  overcome  the  impasse,  and  the  tutor  offering  guidance  and 
confirmations. 

Insert  Table  13  about  here 
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There  is  another  path  through  the  error  recovery  process  illustrated  in  Figure  5.  This 
one,  reminiscent  of  the  emphasis  of  Moore  and  Swartout  (1989)  upon  student  difficulties 
understanding  tutorial  feedback,  consists  of  an  error,  subsequent  feedback,  an  SPSA,  some 
tutorial  guidance,  a  student  confirmation,  and  then  additional  error  feedback.  The  interest¬ 
ing  part  of  this  path  is  that  error  feedback  follows  a  student  confirmation.  This  has  to  do 
with  the  nature  of  our  coding  scheme.  For  example,  in  one  case,  the  tutor  said  You  need 
the  null  case  first,5’  which  would  be  an  example  of  Tutor  Error  Feedback.  If  the  student  did 
not  understand  this,  the  confirmation  would  be  slightly  delayed  relative  to  the  comment 
such  as  a  one-second  pause  before  saying  “Umm,  OK.”  People  in  conversation  are  very 
aware  of  even  very  short  delays  (cf.,  Fox.  1991),  so  a  tutor  might  interpret  this  delay  as 
indicative  of  student  confusion,  and  thus  offer  more  feedback  to  enable  the  student  to  repair 
the  error,  such  as  by  saying  “Since  nil  is  also  a  list,  you  need  to  test  for  null  before  using 
the  listp  test.” 

These  analvses  have  shown  that  students  do  in  fact  contribute  to  error  recovery,  even 
though  tutors  often  initiate  the  process.  Students  and  tutors  together  develop  and  imple¬ 
ment  a  solution  plan,  and  each  may  offer  suggestions  while  the  plan  is  being  implemented. 

Generality  of  the  Results 

In  the  methodology  of  this  study,  we  have  chosen  to  emphasize  the  depth  and  length  of 
interaction  between  relatively  few  tutors  and  several  students  rather  than  a  more  shallow 
analysis  of  more  student-tutor  pairs.  This  depth  of  interaction  has  allowed  us  to  observe 
tutorial  responses  to  a  wide  variety  of  situations. 

Yet.  these  interactions  have  taken  place  in  one  particular  tutorial  setting.  These  stu¬ 
dents  were  very  bright  and  highly  motivated  to  learn  the  material,  and  had  not  previoush 
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experienced  difficulty  with  the  domain,  as  would  be  the  case  in  a  remedial  tutoring  session. 
The  domain  involves  acquiring  proficiency  by  solving  many  exercises,  like  other  mathemat¬ 
ical  and  science  domains,  although  the  cognitive  complexities  of  the  domains  vary.  In  the 
programming  domain,  difficulties  often  arise  as  students  attempt  to  express  their  intentions 
in  a  formal  notation  (Merrill  &  Reiser.  1993).  Tutorial  behavior  is  almost  certainly  affected 
by  a  variety  of  dimensions,  including  the  features  of  the  student  and  the  domain,  and  thus 
the  generality  of  any  individual  study  must  be  examined  carefully.  We  shall  argue,  how¬ 
ever,  that  our  results  do  capture  general  aspects  of  tutorial  behavior  applicable  to  a  wider 
range  of  domains  than  simply  computer  programming.  We  argue  this  in  two  ways,  first,  by 
showing  how  similar  the  two  tutors  were  to  each  other  and  then  by  pointing  out  similarities 
between  our  tutors  and  those  described  in  the  literature. 

First,  the  two  tutors  were  very  similar  to  each  other.  For  example,  recall  that  we  argued 
that  tutors  frequently  responded  to  a  correct  student  action  with  a  Tutor  Confirm  Step  or 
Tutor  Guidance.  The  tutorial  response  to  a  correct  student  action  is  one  indicator  of  tutorial 
style,  since  different  styles  will  lead  tutors  to  respond  to  student  actions  differentially.  If 
these  tutors  display  differing  styles,  then  the  number  of  cases  where  the  tutors  offered  Tutor 
Guidance  versus  those  where  Tutor  Confirm  Steps  were  offered  will  vary.  The  female  tutor 
offered  Tutor  Guidance  about  14%  as  often  as  Tutor  Confirm  Steps.  The  male  tutor  offered 
Tutor  Guidance  in  13%  of  these  cases.  This  suggests  that  the  tutors  are  in  fact  behaving 
similarly.  One  can  calculate  a  more  general  statistic  to  capture  the  degree  to  which  the 
tutors  produced  similar  responses  to  similar  situations  by  comparing  the  interaction  tables 
(Castellan.  1979).  This  technique  essentially  counts  the  situations  in  which  each  tutor  gave 
the  same  category  of  response  to  the  same  student  utterance  type,  and  then  adjusts  that 
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value  for  the  agreement  that  would  be  due  to  chance,  producing  a  chi-square  statistic  to  test 
for  homogeneity  of  the  tutor-student  interactions.  Our  two  tutors  behaved  in  very  similar 
manners  towards  the  students.  x~  <  No  pedagogical  strategies  were  presented  to  the 

tutors,  so  this  similarity  suggests  that  experienced  tutors  may  share  certain  behaviors  when 
examined  over  periods  of  tutoring  substantial  enough  to  allow  a  range  of  problem  solving 
situations  to  occur. 

In  addition  to  being  similar  to  one  another,  our  tutors  also  engaged  at  one  time  or 
another  in  most  of  the  behaviors  previous  theorists  have  highlighted.  For  example,  our 
tutors  offered  confirmations,  as  Fox  (1991)  has  argued,  and  offered  directive  error  feedback 
like  the  tutors  of  Littman  et  al.  (1990)  and  Schoenfeld  et  al.  (1992).  Furthermore  the 
students  and  tutors  worked  together  to  repair  errors  as  Fox  (1991)  has  suggested  and 
this  collaboration  presumably  enabled  students  to  feel  like  they  had  mastered  the  problem 
solving  difficulties  as  Lepper  et  al.  (1990)  has  argued.  Thus,  our  tutors  were  similar  to  one 
another  and  exhibited  at  various  times  the  behaviors  found  in  other  tutorial  studies  in  other 
domains.  Although  characteristics  of  the  students’  abilities  and  the  particular  character  of 
the  domain  may  certainly  influence  the  frequency  and  implementation  strategies  for  these 
tutorial  behaviors,  the  similarity  in  our  tutors  and  the  presence  of  the  range  of  tutorial 
behaviors  suggests  that  we  have  uncovered  some  general  factors  tying  tutorial  behaviors  to 
particular  problem  solving  situations  that  may  occur  in  expert  tutoring  behaviors  in  a  range 
of  problem  solving  domains.  In  the  next  section,  we  present  a  theory  of  tutorial  guidance 
that  attempts  to  describe  why  the  tutorial  behavior  we  found  should  lead  to  pedagogical 
advantages. 
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A  theory  of  tutorial  guidance 

Tutors  help  keep  problem  solving  productive  and  maximize  the  learning  outcomes  oi 
their  students  by  encouraging  and  supporting  successful  problem  solving  and  by  providing 
feedback  to  help  students  refcover  from  errors.  In  this  section,  we  review  the  main  findings 
of  this  study  and  present  a  model  of  tutorial  guidance  that  accounts  for  these  results. 

During  problem  solving,  students  encounter  impasses  and  make  errors.  However,  some¬ 
times  these  errors  are  not  visible  until  many  moves  after  the  error  occurred.  For  example, 
opening  a  chess  game  by  moving  the  rook’s  pawn  forward  one  space  may  seem  reasonable  to 
a  novice  chess  player,  and  the  consequences  of  this  poor  choice  will  not  be  apparent  for  some 
time.  How  is  a  problem  solver  to  understand  which  one  of  many  moves  was  responsible  for 
the  poor  outcome?  Without  this  knowledge,  the  problem  solver  cannot  avoid  the  error  in 
the  next  game.  This  difficulty  is  often  called  the  credit  assignment  problem.  The  credit 
assignment  problem  occurs  when  a  success  or  failure  arises  after  several  steps;  a  learner 
has  to  figure  out  which  step  led  to  the  outcome  experienced.  This  may  be  a  simple  task  if 
there  are  a  small  number  of  steps:  however,  most  classes  of  problems  have  more  than  a  few 
steps  intervening  between  an  event  and  the  associated  marker  of  success  or  failure,  yielding 
a  large  search  space. 

Clearly,  credit  assignment  could  be  facilitated  by  minimizing  the  number  of  steps  be¬ 
tween  signals  of  success  or  failure  and  more  focus  on  the  features  potentially  responsible. 
So.  for  example,  a  student’s  learning  could  be  helped  if  the  student  could  easily  determine. 

after  each  step,  whether  it  was  correct  or  not. 

Anderson  and  his  colleagues  (Anderson  et  al..  1985;  Anderson.  Boyle,  Corbett,  Lewis. 
1990)  have  argued  for  immediate  feedback  as  an  effective  pedagogical  technique  in  problem 
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solving  domains  due  to  the  difficulties  that  error  recovery  can  pose  for  learning  (Anderson. 
Conrad,  Sz  Corbett,  1989).  Anderson  and  Corbett  (1993)  argue  that  immediate  feedback 
may  not  lead  to  superior  pedagogical  outcomes,  but  does  lead  to  more  efficient  learning. 
That  is.  students  receiving  immediate  feedback  can  learn  a  domain  equally  well  as  students 
without  the  feedback,  but  can  do  so  up  to  three  times  faster. 

In  fact,  our  tutors  engaged  in  behaviors  that  minimized  credit  assignment  to  a  few  events 
via  Tutor  Confirm  Steps  and  Tutor  Error  Feedback.  Our  tutors  responded  to  both  correct 
and  incorrect  steps  very  rapidly,  sometimes  even  on  the  next  event,  thus  minimizing  the 
space  of  possible  actions  that  could  have  lead  to  an  error  or  success.  The  Tutor  Guidance 
further  helped  encourage  students  on  promising  solution  paths  and  helped  focus  their  search 
when  necessary.  Furthermore  the  very  interactive,  efficiently  communicated  nature  of  this 
feedback  means  that  students  could  respond  to  the  feedback  without  unduly  interrupting 
their  problem  solving  effort.  This  focusing  of  students’  search  appears  to  be  a  central  source 
of  pedagogical  advantage  for  individualized  instruction. 

The  tutors  responded  to  different  types  of  errors  with  different  feedback  strategies.  We 
next  turn  to  a  discussion  of  the  pedagogical  benefits  of  this  practice. 

There  is  a  trade-off  in  learning  from  errors.  With  each  error  there  are  benefits  of  self- 
recovery.  Students  learn  more  when  they  construct  explanations  for  themselves  rather  than 
simply  encoding  a  provided  explanation  (Chi  et  al.,  1989).  However,  the  costs  of  floundering 
in  time,  confusion,  and  frustration  can  be  serious,  especially  if  students  do  not  construct 
explanations.  Indeed,  the  benefit  of  self-explanation  has  been  primarily  demonstrated  in 
students  studying  instructional  examples  rather  than  interspersing  self-explanations  in  their 
own  problem  solving.  We  suggest  that  tutorial  response  to  errors  can  be  explained  by 
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comparing  the  relative  weight  of  the  learning  opportunity  s  potential  benefits  to  its  potential 
costs.  We  call  this  relative  weighting  the  learning  consequences  of  an  error.  By  examining 
the  learning  consequences  of  each  of  the  three  types  of  errors,  we  will  see  that  the  relative 
weights  of  costs  and  benefit^  predict  the  type  of  feedback  tutors  provided.  When  there 
was  a  great  deal  of  learning  possible  from  self-recovery,  tutors  allowed  students  to  perform 
as  much  of  the  error  recovery  as  possible.  However,  if  the  costs  of  repair  outweighed  the 
benefits,  tutors  simply  told  the  student  how  to  repair  the  error,  thus  keeping  the  student 
on  track  to  a  solution. 

Recall  that  there  were  three  types  of  student  errors:  syntactic,  semantic,  and  goal  errors. 
First  consider  syntactic  errors.  The  syntax  of  computer  programming  languages  is  a  source 
of  difficulty  for  novices,  and  novices  often  spend  a  great  deal  of  time  flailing  around  trying  to 
fix  errors  arising  from  mistakes  in  the  notational  syntax  (Anderson  et  ah,  1985;  du  Boulay, 
1986;  Reiser  et  ah,  1991). 

Programming  language  syntax  is  often  a  source  of  great  difficulty  for  novices  for  a  variety 
of  reasons.  Among  these  reasons  are  that  language  syntax  is  often  arbitrary  and  has  little 
relation  to  the  ways  novices  think  about  real-world  analogues  of  the  constructs  (Bonar  <k: 
Soloway,  1985;  Trafton  &  Reiser,  1993b).  Further,  novices  are  used  to  having  some  margin 
for  error  in  communication  in  the  real  world,  and  sometimes  fail  to  consider  exactly  how 
literal  one  must  be  when  creating  a  computer  program  (Bonar  &;  Soloway,  1985,  du  Boula\, 
1986;  Pea,  1986).  Due  to  the  largely  arbitrary  nature  of  syntactic  rules,  resolving  errors 
typically  relies  on  weak  method  search  or  analogy  from  examples,  not  from  explanation. 
Thus,  students  may  learn  little  more  by  repairing  errors  themselves  than  by  simply  being 
told  how  to  do  the  repair.  Furthermore,  the  difficulty  of  repairing  syntactic  errors  creates 
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a  serious  danger  of  floundering.  Thus,  syntactic  errors  have  low  learning  consequences. 
Accordingly,  tutors  usually  just  stepped  in  and  offered  a  Tutor  Correction  that  told  the 
student  how  to  repair  the  error  directly,  requiring  the  student  only  to  implement  the  repair. 
This  feedback  strategy  sacriijces  the  few  benefits  that  might  accrue  from  the  repair  in  favor 
of  keeping  the  student  on  a  productive  repair  path,  and  since  there  is  little  of  the  recovery 
process  remaining  for  the  student  to  do,  recovering  from  these  errors  is  quite  rapid. 

In  a  Semantic  Error,  the  student  misapplied  a  LISP  function  or  basic  concept.  How 
should  tutors  support  student  reasoning  after  this  sort  of  error?  Semantic  Errors  usually 
received  Surface  Feature  Feedback  that  pointed  to  the  incorrect  feature  evident  in  the 
solution.  Given  how  much  difficulty  novices  have  locating  and  repairing  recognizing  errors 
in  programs  (Katz  &  Anderson,  1988;  Reiser  et  ah,  1991)  and  the  working  memory  load 
caused  by  the  search  that  interferes  with  learning  (Anderson  et  ah,  1985;  Sweller,  1988),  one 
might  expect  tutors  to  simply  intervene  with  feedback  that  performs  all  of  the  components 
of  the  error  recovery  process  in  the  interest  of  keeping  students  from  becoming  overloaded, 
as  was  the  case  with  syntactic  errors.  However,  a  critical  component  of  the  learning  in 
a  new  domain  involves  acquiring  the  semantics  of  operators  and  reasoning  about  their 
interactions  when  learning  to  construct  more  complex  plans  involving  the  operators  (Merrill 
&  Reiser,  1993;  Ohlsson  &  Rees,  1991;  Solowav,  1986).  Furthermore,  repairing  errors  often 
involves  reasoning  about  causes  and  consequences  of  the  observed  erroneous  behavior  (Katz 
Sz  Anderson,  1988;  Spohrer.  Soloway,  &;  Pope,  1985),  and  successful  students  generate  these 
sort  of  explanations  in  other  problem  solving  situations  (Chi  et  al..  1989),  so  self  generated 
explanations,  with  tutorial  collaboration  if  necessary,  seems  a  promising  mechanism  for 
students  to  acquire  this  knowledge. 
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Since  the  tutor  only  pointed  the  student  to  the  general  location  of  the  error,  the  student 
had  to  recognize  what  was  incorrect  in  the  solution,  make  an  inference  about  the  error  s 
nature,  set  a  goal  to  repair  the  error,  and  then  begin  to  implement  the  repair.  Thus, 
students  were  able  to  learn  a£>out  the  domain  operators  through  a  very  focused  learning  b\ 
doing  session  involving  recovering  from  an  error.  This  additional  problem  solving  effort  was 
reflected  in  the  longer  error  recovery  episodes.  The  tutors  participated  in  the  entire  recover} 
process  as  well,  serving  to  keep  the  student  from  dangerous  floundering  (cf.,  McArthur  et  ah, 
1990),  and  rendering  the  error  recovery  work  profitable. 

The  third  class  of  errors  concerned  the  students  manipulation  of  the  goal  structure. 
Structuring  behavior  according  to  goals  is  a  critical  feature  of  learning  new  problem  solv¬ 
ing  domains  (Anderson.  1983;  Newell,  1990;  VanLehn.  1990).  Indeed,  many  instructional 
theorists  have  argued  that  helping  students  maintain  and  reflect  on  a  goal  structure  facil¬ 
itates  learning  a  new  domain  (Collins  &  Brown,  1988;  Collins,  Brown,  Newman,  1989, 
Koedinger  &  Anderson,  1990;  McArthur  et  ah,  1990;  Merrill  &  Reiser,  1993;  Singley,  1990). 

Recall  that  goal  errors  occurred  when  the  student's  manipulation  of  the  goal  structure 
faltered.  How  did  tutors  offer  the  required  support?  A  variety  of  feedback  types  occurred 
on  these  errors.  Surface  Feature  Feedback,  as  on  the  semantic  errors,  helped  students 
pinpoint  the  location  in  the  solution  where  their  reasoning  went  awry  and  reconstruct  a 
more  promising  solution  plan.  In  other  cases,  the  support  took  the  form  of  a  Tutor  Plan- 
Based  Feedback,  reminding  the  student  what  the  current  goal  should  be.  This  feedback  also 
told  the  student  of  the  location  of  the  error.  However,  in  a  Goal  Error,  the  relevant  location 
was  not  an  explicit  component  of  the  soiution.  but  rather  a  subgoal  whose  implementation 
was  faulty  or  forgotten.  Thus,  locating  the  error  requires  referring  to  the  more  abstract  goal 
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structure,  hence  the  appropriateness  of  Tutor  Plan-Based  Feedback.  Although  the  tutor 
can  of  course  offer  additional  help  throughout  the  repair,  this  initial  feedback  alerts  the 
student  to  the  existence  of  an  error  and  gives  some  information  about  its  location,  with  the 
tasks  of  realizing  what  has  b^en  implemented  incompletely,  setting  the  goals  to  perform  the 
repair,  and  actually  implementing  it  remaining  to  be  accomplished.  Thus,  tutors  support 
students5  maintenance  of  a  goal  structure  in  a  similar  manner  to  their  support  of  difficulties 
with  operators,  by  pointing  students  to  a  location  in  a  structure  that  will  highlight  the 
error.  This  allows  students  the  opportunity  to  perform  much  of  the  error  feedback,  as  in 
semantic  errors,  but  prevents  potential  floundering. 

Our  analyses  have  shown  that  tutors  modulate  their  feedback  in  accordance  with  learn¬ 
ing  consequences.  Examining  the  errors  that  led  to  our  three  types  of  tutorial  feedback 
showed  that  tutors  tended  to  give  explicit  corrections  that  contained  directions  for  the  er¬ 
ror’s  repair  when  an  error  did  not  offer  the  opportunity  for  significant  learning  but  could 
lead  to  unfruitful  floundering.  Those  low  learning  consequences  errors  offer  few  benefits 
of  learning  by  doing,  and  thus  tutors  simply  tell  the  student  how  to  repair  the  error,  and 
the  students  do  so.  In  contrast,  when  it  would  be  beneficial  for  the  student  to  explain  the 
erroneous  part  of  a  solution  and  plan  the  repair,  such  as  in  semantic  errors,  tutors  focus 
students  on  the  error  but  let  them  replan  a  solution  for  that  goal.  When  the  target  concept 
is  more  abstract,  such  as  understanding  the  goal  structure,  tutors  may  leave  the  analysis 
of  the  error’s  feature  to  the  student  as  well.  Because  of  these  strategies,  students  get  to 
learn  to  recover  from  errors  by  doing,  thus  actively  exercising,  testing,  and  modifying  their 
problem  solving  knowledge,  but  also  get  protected  from  some  of  the  more  severe  costs  of 
the  recovery  process  by  carefully  chosen  tutorial  feedback.  This  balance  of  learning  by 
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doing  and  tutorial  guidance  maximizes  the  effectiveness  of  students  problem  solving  and 

underlies  the  strong  learning  gains  for  tutored  students. 

In  this  paper,  we  have  presented  a  model  of  the  relationship  between  student  errors  and 
feedback,  learning  consequences,  and  also  described  how  tutors  support  ongoing  problem 
solving  through  confirmations  and  rapid  error  flagging.  We  now  return  to  the  views  of 

tutoring  advocated  by  the  researchers  reviewed  earlier. 

Fox  (1991)  argued  that  confirmatory  feedback  serves  a  central  role  in  tutorial  discourse. 
Our  results  support  this,  and  we  argued  that  the  confirmations  support  problem  solving 
by  enabling  students  to  determine  more  easily  which  action  was  responsible  for  success 
and  which  knowledge  was  faulty.  Lepper  and  his  colleagues  (Lepper  et  al.,  1990,  Lepper  it 
Chabay,  1988)  found  that  tutorial  feedback  served  to  help  students  remain  feeling  successful, 
and  that  this  accounts  for  tutorial  pedagogical  benefits.  In  fact,  we  found  a  relative  lack 
of  feedback  specifically  motivational  in  character  in  our  data.  Presumably,  this  reflects  the 
differences  in  ages  and  domains  between  the  two  studies.  This  reinforces  the  importance 
of  considering  the  multidimensionality  of  tutorial  behavior,  since  changes  in  some  of  these 
dimensions  led  us  to  find  almost  no  instances  of  motivational  behaviors  that  Leppers  tutors 
found  very  important.  Lepper  s  tutors  were  teaching  small  children  who  had  already  had 
difficulty  mastering  the  arithmetic  material.  This  social  situation  seems  destined  to  brin0 
out  the  supportive  side  of  any  compassionate  instructor.  Our  students  were  bright,  confident 
college-aged  students  learning  a  domain  for  the  first  time,  who  presun  ably  required  les^ 
motivational  support.  In  fact,  we  suggest  that  our  tutors  did  achieve  positive  motivational 
benefits  but  did  so  by  minimizing  student  frustrations  and  helping  to  keep  problem  solvin0 
productive  (cf..  Reiser  et  al..  1994).  rather  than  by  directly  reinforcing  students’  feeling  of 
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success  or  by  helping  them  explain  away  difficulties  encountered. 

Our  analyses  of  tutorial  guidance  have  suggested  that  part  of  tutorial  effectiveness 
relies  on  tailoring  feedback  to  particular  problem  solving  situations.  Indeed,  the  ability 
to  individualize  guidance  to^particular  contexts  has  been  proposed  as  a  central  advantage 
of  individualized  instruction  (Bloom.  1984:  Cohen  et  al.,  1982;  McArthur  et  ah,  1990) 
and  has  motivated  much  work  in  computerized  intelligent  tutoring  systems  (e.g.,  Anderson 
et  ah.  1985;  Carbonell,  1970;  Goldstein.  1982).  In  contrast,  several  studies  of  tutoring 
have  suggested  that  tutors  follow  a  fairly  uniform  simple  sequence  of  pedagogical  actions 
while  tutoring  students,  essentially  following  straightforward  curriculum  scripts,  in  which 
they  present  and  query  topics,  backing  up  and  spending  more  time  as  needed  to  cover 
troublesome  ones  (Graesser,  1993;  Putnam,  1987;  Sleeman,  Kelley,  Martinak,  Ward,  & 
Moore,  1989).  In  this  view  of  tutoring,  curriculum  goals  that  do  not  vary  across  students 
account  for  most  tutorial  actions. 

Our  results  suggest,  at  least  for  tutoring  sessions  focusing  on  problem  solving,  that 
this  curriculum  script  model  is  an  incomplete  description  of  student-tutor  interactions. 
We  found  that  student-tutor  dialogues  were  centered  much  more  around  student-initiated 
events,  as  they  attempted  to  actively  understand  new  instructional  material  and  solve 
problems,  than  around  tutorial  presentation  of  material  and  subsequent  querying  of  stu¬ 
dent  understanding.  These  more  complex  patterns  of  student-tutor  interactions  are  better 
described  by  a  mixed-initiative  dialogue  (Carbonell,  1970;  Collins  et  al.,  1975)  than  the 
dialogue  frames  suggested  by  Graesser  (1993)  or  the  curriculum  scripts  suggested  by  Put¬ 
nam  (1987).  The  tutors  responded  with  guidance  to  support  and  focus  students  reasoning, 
rather  than  presenting  material  themselves  and  then  querying  student  understanding. 
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A  central  issue  in  this  type  of  tutorial  guidance  concerns  how  tutors  responded  to  student 
errors.  As  Putnam  (1987)  and  Lepper  and  Chabay  (1988)  have  pointed  out,  effective  tutors 
do  not  appear  to  attempt  to  diagnose  the  faulty  reasoning  that  caused  the  students  errors. 
For  example,  tutors  do  not  pippose  new  problems  for  students  to  solve  purely  for  the  purpose 
of  discriminating  between  possible  misconceptions  that  might  have  caused  the  error.  Tutors 
also  rarelv  relate  possible  lines  of  student  reasoning  that  could  have  led  to  the  error.  Indeed. 
Sleeman  and  his  colleagues  have  argued  that  pointing  out  the  students'  misconception  to 
the  student  is  no  more  effective  than  simply  reteaching  the  erroneous  procedure  (Sleeman 
et  ah.  1989;  Sleeman,  Ward,  Kelly,  Martinak,  k  Moore,  1991),  which  might  suggest  that 
diagnosis  is  unnecessary.  In  these  views,  effective  tutorial  response  to  errors  consists  of 
simply  reteaching  the  correct  procedure  rather  than  focusing  on  explanations  of  why  errors 
occurred. 

Our  microanalyses  of  the  student-tutorial  interactions  in  problem  solving  situations 
suggest  that  tutors  do  more  than  simply  reteach  a  correct  procedure  component  when 
students  encounter  impasses  or  errors.  Our  tutors  focused  on  guiding  the  error  repair 
process  rather  than  on  communicating  their  guesses  about  the  student  s  misconception. 
As  Merrill  et  al.  (1992)  suggested,  whether  tutors  verbalize  diagnoses  is  a  less  central 
question  than  whether  (and  how)  they  track  student  reasoning  and  determine  what  type  of 
guidance  to  provide.  The  results  of  our  present  study  demonstrate  the  ways  in  which  tutors 
focus  students  on  detecting  and  repairing  errors.  Tutors  do  not  simply  correct  students 
and  review  relevant  curriculum  material.  Instead,  they  collaboratively  help  the  students 
understand  and  repair  errors.  Furthermore,  tutors  tailor  the  timing  and  specific  content  of 
their  feedback  to  the  learning  consequences  of  the  particular  error.  Our  results  demonstrate 
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that  tutors  intervene  less  quickly  and  leave  more  of  the  error  repair  to  students  when  more 
can  be  learned  from  the  error  repair.  Thus,  rather  than  a  path  through  a  curriculum  script 
or  dialogue  frame,  we  see  very  careful  tracking  of  student  reasoning  and  modulation  of  the 
timing  and  nature  of  feedback  depending  on  the  type  of  error  encountered.  However,  we 
should  note  that,  consistent  with  the  reteaching  view,  the  principal  goal  seems  to  getting  the 
problem  solving  back  on  track.  These  errors  episodes  in  general  were  very  short  (typically 
two  or  three  events)  and  were  always  focused  on  repairing  the  error,  rather  than  exploring 
it. 

The  careful  adaptation  of  feedback  timing  and  content  suggests  that  a  more  complex 
model  than  curriculum  scripts  is  needed  to  explain  tutorial  behavior.  We  suggest  that 
tutorial  behavior  is  better  modeled  by  microplans  (McArthur  et  ah,  1990;  Schoenfeld  et  ah, 
1992)  in  which  particular  tutorial  plans  are  triggered  in  response  to  particular  student 
problem  solving  situations.  Our  analyses  suggest  that  the  central  microplans  for  modeling 
tutorial  behavior  must  track  student  reasoning  and  determine  when  to  provide  confirmations 
or  additional  guidance  upon  correct  paths,  and  also  specify  when  to  offer  feedback  to  student 
errors  and  how  much  of  the  error  recovery  process  to  perform  after  this  intervention. 

Conclusion 

In  this  study,  we  used  extensive  analysis  of  many  hours  of  student-tutor  discourse  to 
attempt  to  determine  the  strategies  experienced  human  tutors  use  that  result  in  the  peda¬ 
gogical  advantages  of  human  tutoring.  These  analyses  suggest  that  tutors  assist  students 
active  problem  solving  with  careful  guidance,  in  which  the  tutor  keeps  the  student  s  problem 
solving  on  track  via  ongoing  confirmatory  feedback  and  new  goals  to  achieve  after  correct 
steps  and  error  feedback  after  errors.  Students  caught  some  of  their  own  errors,  but  if  the 
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student  did  not  notice  an  error  had  occurred,  the  tutor  drew  the  student's  attention  to  it 
relatively  quickly. 

We  argued  that  the  relative  weights  of  the  benefits  of  self-repair  versus  the  costs  of 
floundering  dictated  the  feedback  used  by  tutors.  In  situations  where  an  error  could  lead 
to  floundering  and  did  not  offer  significant  potential  for  learning,  tutors  often  told  the 
student  how  to  repair  the  error,  thereby  leaving  only  the  implementation  of  the  repair  to 
the  student.  In  contrast,  tutors  offered  less  support  for  errors  that  offered  significant  benefits 
of  self-repair.  Thus,  as  learning  consequences  increased,  the  tutors  allowed  the  students  to 
perform  more  and  more  of  the  error  recovery,  including  constructing  their  own  explanations 
for  the  errors  and  acting  on  their  analyses  (cf.,  Chi  et  al.,  1989)  This  active  self-explanation 
and  problem  solving  leads  students  to  develop  better  models  of  the  behavior  of  operators 
in  the  domain. 

Tutorial  guidance  allows  an  extremely  effective  style  of  learning  by  doing  guided 
learning  by  doing.  Students  can  pursue  the  benefits  of  actively  constructing  understandings 
and  solution  plans  and  implementing  them  with  carefully  modulated  guidance  from  the 
tutor.  Tutors  modulate  their  guidance  in  response  to  students'  actions  and  current  problem 
solving  context.  Tutors  encourage  students  to  continue  on  profitable  paths,  and  warn 
students  of  errors  through  explicit  and  rapid  comments  that  focus  students’  attention  on 
sources  of  errors.  When  obstacles  are  encountered,  students  and  tutors  collaboratively 
effect  a  repair.  During  this  repair,  tutors  offer  guidance  and  feedback  but  do  so  in  a 
manner  that  encourages  students  to  analyze  the  error  and  actively  contribute  to  the  repair. 
This  carefully  modulated  guidance  allows  the  best  pedagogical  advantages  of  learning  by 
doing  while  minimizing  the  consequent  potential  costs  of  self-directed  search  for  a  correct 
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answer  through  a  very  large  problem  space.  This  careful  tutorial  guidance  offered  during 
successful  problem  solving  as  well  as  during  difficulties  leads  tutored  students  to  achieve 
the  substantial  cognitive  and  motivational  advantages  observed  in  individualized  tutoring. 
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Appendix  A:  Instructions  for  Coders  and  Reliability  Coders 

Segmentation  Instructions 

1.  Break  a  new  segment  if  and  only  if  you  can  clearly  demonstrate  a  need  to  do  so.  In 

< 

other  words,  a  segment  should  be  continued  until  a  concept  is  introduced  that  was 
not  present  before. 

2.  Pronouns  are  tricky.  If  you  are  segmenting  and  run  across  a  new  usage  of  some 
pronoun  fe.g.,  “it”),  try  to  find  the  meaning  of  the  pronoun  in  the  current  segment. 
Only  break  a  segment  when  a  pronoun  is  introduced  if  the  pronoun  cannot  be  bound 
to  a  noun  in  the  current  segment. 

3.  Segments  can  continue  across  interruptions  if  the  interruption  is  not  heeded  by  the 
speaker.  If  something  is  said  as  an  interruption  (no  matter  how  small)  that  the 
speaker  responds  to  in  any  way ,  a  new  segment  must  be  made  for  the  interruption. 

4.  Segmentation  based  solely  upon  typing  must  be  done  very  carefully,  because  there  is 
very  little  information  in  a  typing  episode.  Thus,  a  segment  can  be  made  if  the  student 
completes  a  complete  LISP  action  (defun,  function  call)  or  if  there  is  conversation 
during  the  typing,  but  never  within  a  defun. 

5.  Segmentation  is  independent  of  categorization.  Don’t  worry  about  what  a  segment 
will  be  -  break  according  to  the  rules  above. 

Coding  Instructions 

1.  Each  segment  must  receive  1  and  only  1  of  the  codes  that  appear  in  the  attached 
pages. 
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2.  Categorize  based  upon  what  was  said,  not  what  you  think  the  utterance  meant.  The 
codes  axe  grouped  into  questions  and  assertions.  If  you  are  categorizing  a  question, 
the  code  you  apply  must  be  one  of  the  question  codes,  etc. 

3.  The  default  categories <are  marked.  Unless  you  have  definite  reason  to  do  otherwise, 
you  should  use  the  default  category. 

4.  Do  not  “read  into”  category  definitions.  If  an  utterance  does  not  quite  fit  category 
A.  it  does  not  fit!  Do  not  stretch  the  categories  to  fit  an  utterance. 

5.  Sometimes  a  typing  episode  will  be  in  the  middle  of  a  segment.  Encode  this  segment 
according  to  the  most  important  element.  For  example,,  if  the  person  is  simply  saying 
what  they  are  typing,  the  segment  should  be  categorized  as  typing,  and  so  forth. 
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Appendix  B:  Definitions  of  Coding  Categories 

The  categories  are  grouped  here  as  they  were  in  Tables  3  and  4.  All  examples  are  taken 
verbatim  from  the  actual  protocols. 

The  student  constructs  a  solution  (Student  Problem  Solving  Action) 

Student  Correction :  This  category  consists  of  self-corrections  by  the  student,  in  which 
the  student  has  made  an  error,  but  immediately  recognizes  it  and  suggests  how  to  fix  it. 

Oh.  I  need  a  quote  before  that! 

Whoops,  I  need  to  add  a  parenthesis  there. 

Student  Elaboration:  An  utterance  is  categorized  as  an  Elaboration  if  the  student  is 
answering  a  question,  without  providing  actual  data  (which  would  be  a  Student  Simulate 
Process,  see  below),  or  is  simply  adding  to  the  information  already  presented  in  the  con¬ 
versation.  Thus,  Student  Elaboration  is  a  default  category  for  student  to  tutor  assertions 
that  are  task  related,  but  do  not  fit  any  other  category. 

OK,  an  empty  list. 

The  rest  of  the  sentence  sifter  by  . . . 

Student  Example :  The  student  produces  a  concrete  example  to  demonstrate  some  point, 
or  to  ask  a  question. 

What  about  “a”  and  “(b  c  d) ”? 

What  about  nil  then? 
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Student  Focus  Attention:  The  student  causes  some  item  in  the  book  or  on  the  screen  to 
become  the  focus  of  the  conversation. 

And  this  cond  is  the  thing  we're  finding. 

Oh.  they’re  talking  about  this! 

Student  Indicate  Difficulty:  The  student  remarks  that  a  problem  is  difficult  (or  long, 
and  so  forth),  or  makes  some  comment  indicating  that  he  or  she  thinks  the  next  section 
will  be  hard. 

Writing  this  would  be  a  problem. 

Boy  this  is  long! 

Student  Indicate  Lack  of  Understanding :  These  utterances  tell  the  tutor  that  the  student 
is  confused.  This  could  be  done  directly,  or  indirectly,  such  as  through  several  repetitions 
of  the  same  word,  without  making  any  progress  toward  answering  the  question  (as  in  the 
second  example). 

I  don't  understand  what  they  mean  here. 

Oh,  parameters,  parameters  are  . . .  umm  . . .  they  re  uhh  . . . 

Student  Read:  The  student  reads  from  the  textbook.  This  is  marked  in  the  transcripts 
by  markers  such  as  “[2.1]”.  The  numbers  reflect  the  section  of  the  text  being  read. 

Student  Refer:  The  student  refers  to  other  material  to  shed  light  on  the  current  situation. 
The  material  could  be  a  previous  problem,  a  section  of  the  text  that  was  alread>  read. 
The  important  aspect  is  that  the  student  uses  previous  work  to  cast  light  on  the  current 
situation. 
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This  is  just  like  what  we  did  yesterday,  the  pal  thing. 

Actually,  this  would  have  been  true  if  this  is  greater  than.  It’s  just  the  same 
thing. 

Student  Set  Goal:  The  student  sets  a  goal  for  what  to  do  next,  or  indicates  how  to  do 
the  next  step.  Thus,  this  category  includes  both  statements  about  goals  and  the  plans  that 
can  achieve  them. 

So  they  want  us  to  write  down  for  each/ / 

Now  I  have  to  put  the  quote. 

Student  Type :  The  student  types  into  the  LISP  interpreter,  or  writes  on  the  paper. 
This  is  denoted  by  either  “(writing)”  or  by  the  time  the  student  began  typing,  such  as 
“[20:26:23]”. 

[20:10:45] 

Then  this,  (writing) 


The  student  asks  for  help  from  the  tutor 

Assist  Plan  Assertion:  This  category  contains  utterances  that  request  an  evaluation  of 
the  student’s  plan  or  understanding  of  the  problem. 

But  we  can’t  do.  can’t  we  do  cons  twice  or  something? 

Do  I  ...  do  I  go  zerop  nurril 
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Assist  Plan  Question:  This  category  consists  of  utterances  that  are  requests  for  the  tu¬ 
tor’s  help  in  deciding  what  to  do  next.  These  utterances  can  be  implied  or  actual  questions. 

Now  I  should  do  . . . 

I  don't  know  what  to  do  now. 

Assist  Understanding:  This  category  contains  utterances  that  ask  for  the  tutor  to  eval¬ 
uate  the  student’s  understanding  of  a  LISP  concept.  These  utterances  can  be  questions  or 
implied  questions. 

Do  you  mean  to  say  it  can't  be  more  than  one,  right? 

But  what  do  you  mean  by  variable,  this  is  the  variable? 

Student  Informational  Request:  This  is  the  most  conservative  student  to  tutor  request 
category.  If  there  is  no  reason  to  think  that  a  student  request  is  either  an  Assist  Under¬ 
standing  or  an  Assist  Plan,  then  the  utterance  should  be  coded  as  this.  This  category  also 
includes  requests  such  as  how  to  use  the  editor,  how  to  type  parentheses. 

Exit-with-save  was  what.  F9? 

But  why  are  they  saying  that  it’s  surprising? 


The  student  indicates  that  the  tutor's  utterances  were  understood 

Student  Confirmation:  The  student  says  something  to  indicate  that  he  or  she  is  following 
along  with  what  the  tutor  said,  either  by  some  sort  of  restatement  or  a  simple  OK. 
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It  can  be  very  complex.  I  understand. 

Yes.  this  is  what  we  want. 

The  student  checks  the  current  answer 

Student  Simulate  Process:  This  category  is  similar  to  Elaborate,  with  a  crucial  differ¬ 
ence.  SSP  contains  utterances  that  require  the  student  to  work  through  the  behavior  the 
computer  would  execute  on  the  current  example,  either  verbally  or  nonverbally.  In  other 
words,  utterances  that  describe  how  the  LISP  would  actually  process  some  definition  are 
SSP.  In  addition,  utterances  that  produce  data  output  from  functions  falls  here  as  well, 
since  to  produce  data,  the  student  must  have  “run  the  function'5  in  his  or  her  head. 

And  then  after  that’s  done,  see  I  want  to  put  d  again,  and  again  the  same  thing. 

Oh  this  will  give  me  (plum  apple  cantaloupe  grape). 


Miscellaneous  non-task-related  utterances 

Student  Comment :  This  category  contains  unclassifiable  utterances  and  unrelated  talk. 
In  addition,  if  a  student  makes  an  assertion  that  cannot  be  put  into  another  category 
because  it  is  unclear  what  is  being  said,  the  utterance  is  categorized  here. 

Oh,  hi  there,  just  a  second. 

Well.  you.  this  is// 


77 


Tutoring:  Guided  learning  by  doing 


The  tutor  performs  a  portion  of  the  problem  solving 

Tutor  Example:  This  is  the  tutorial  version  of  Student  Example;  thus,  this  category 
consists  of  the  tutor  proposing  a  concrete  example  to  be  worked  on.  This  is  listed  as  a 
question,  because  these  utterances  often  take  the  form  of  “What  about  list  a  b  c? 

OK.  how  about  the  input  nil  there? 

What  about  using  “(a  bed)”  there? 

Tutor  Focus  Attention:  This  is  the  tutorial  version  of  Student  Focus  Attention,  and 
consists  of  the  tutor  making  something  the  topic  of  conversation,  without  giving  any  new 
information  about  it.  Thus,  the  tutor  could  be  pointing  to  something  in  the  text,  or  to 
something  that  had  just  been  said. 

This  is  remember,  a  variable  now. 

And  defun  always  returns  the  name  of  the  function. 

Tutor  Read:  This  is  the  tutorial  version  of  Student  Read.  The  tutor  reads  from  the 
textbook,  and  is  denoted  by  marker  such  as  “[2.1]”  The  numbers  refer  to  the  section  of  the 
text  that  the  tutor  read. 

Tutor  Refer:  This  is  the  tutorial  version  of  Student  Refer.  These  utterances  involve 
the  tutor  bringing  previous  work  to  bear  on  the  current  situation.  The  work  could  be  a 
previous  part  of  the  textbook,  or  a  previous  problem,  or  even  an  alternate  way  of  solving 
a  problem. 

Just  like  car  and  edr  and  cons  have  names,  if  you're  defining  a  new  function  you 
need  to  give  it  a  name. 
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It’s  just  like  what  you  were  doing  yesterday,  figuring  out  what  you  want  to  do. 

Tutor  Type:  This  is  the  tutorial  version  of  Student  Type,  and  is  either  marked  with  the 

time  the  typing  occurred  or  with  ‘'(writing)'’ . 

< 

[20:10:45] 

Then  this,  (writing) 

The  tutor  offers  guidance  for  the  student’s  ongoing  problem  solving 

Tutor  Confidence  Builder:  The  tutor  expresses  confidence  in  the  student’s  ability  to 
solve  the  problem  or  offers  praise  to  the  student  about  a  specific  problem  solving  success. 

It’s  really  good  that  you  thought  to  put  this  first! 

Yeah,  but  the  last  one  will  be  easy  for  you. 

Tutor  Hint:  This  category  captures  tutorial  utterances  that  hint  at  the  next  step,  but 
do  not  give  it  fully.  Thus,  these  utterances  are  similar  to  Tutor  Set  Goal  (see  below),  but 
are  not  as  directive  —  rather,  they  just  suggest  a  course  of  action  to  be  considered. 

Remember,  you  have  the  less-than  and  greater-than  predicates. 

For  example,  what  about  the  predicate  that  tests  atom ? 

Tutor  Indicate  Difficulty:  This  is  the  tutorial  version  of  Student  Indicate  Difficulty.  This 
category  includes  utterances  that  describe  the  current  problem  as  hard  or  long,  and  so  on. 
or  tell  the  student  that  the  next  set  of  problems  will  be  very  difficult. 
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But  you  see.  this  is  the  complex  part. 

And  I  have  to  warn  you.  these  are  getting  tougher,  so  don  t  worry  if  it  takes 

you  longer. 

Tutor  Set  Goal:  This  is  the  tutorial  version  of  Student  Set  Goal,  and  includes  assertions 
about  what  to  do  next,  or  how  to  do  it.  The  statement  may  refer  to  a  new  goal,  or  to  a 
plan  that  achieves  a  stated  goal. 

So  now.  if  you.  you  could  type  this  in.  We  can  do  some  examples  with  it. 

So  you’ll  need  the  function  name,  then  the  parameters,  and  then  the  body. 

Tutor  Supportive  Statement:  This  category  contains  utterances  which  are  designed  to 
make  the  student  aware  that  the  tutor  is  there  to  help  if  needed. 

And  then,  whenever  you  have  a  question,  just  let  me  know. 

I  can  help  you  if  you  need  it. 

The  tutor  confirms  a  student  step  (TCS) 

Tutor  Confirmation:  This  is  the  tutorial  version  of  Student  Confirmation.  Thus,  these 
utterances  indicate  that  the  tutor  is  following  along  with  what  the  student  is  saying  or 
doing.  This  could  be  done  via  a  restatement  or  a  simple  ‘‘OK.  ’  Notice  that  this  category 
includes  the  tutor  telling  the  student  that  the  step  is  right  or  saying  that  the  student  s  last 
comment  was  understood. 

Right,  uh  huh. 

Yes,  this  part. 
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Tutor  Elaboration:  This  is  the  tutorial  version  of  Student  Elaboration.  Tutor  utterances 
are  categorized  a is  Elaborations  if  the  tutor  answers  a  question,  without  providing  explicit 
data  or  output  of  a  function  (which  would  be  a  Tutor  Simulate  Process,  see  below),  or  is 
simply  saying  something  wh^ch  adds  to  the  information  present  in  the  discussion.  This  is 
a  default  tutorial  utterance,  so  if  a  tutorial  assertion  seems  on  task,  but  does  not  fit  any 
other  category,  it  should  go  here. 

Right,  because  it's  embedded  in  this  bigger  list,  but  when  you  take  it  out.  this 
is  just  like  a  list  with  two  separate  . . . 

And  divide  is  slash  which  is  near  the  question  mark. 


The  tutor  gives  error  feedback  after  an  incorrect  student  step 

Tutor  Correction :  This  is  the  tutorial  version  of  Student  Correction,  and  involves  the 
tutor  telling  the  student  exactly  how  to  fix  an  error.  The  presence  of  a  direct  statement 
of  how  to  fix  the  error  is  the  marker  of  a  Tutor  or  Student  Correction.  The  subject  of  the 
correction  and  the  amount  of  explanation  in  the  correction  can  vary,  but  there  must  be  an 
explicit  direction  for  fixing  the  error. 

And  one  more  for  this  one. 

No.  in  a  list. 

Tutor  Plan-Based  Feedback:  This  is  one  of  the  forms  of  tutorial  error  feedback.  This 
type  of  feedback  requires  that  the  tutor  knows  what  the  student  was  trying  to  do  when 
the  error  occurred.  If  the  student  makes  an  error  and  the  tutor  responds  to  that  error 
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by  referring  the  student  to  the  goal  that  the  student  should  have  been  working  on.  that 
utterance  should  be  categorized  as  a  Tutor  Plan-Based  Feedback.  These  utterances  are 
similar  to  Tutor  Set  Goal  (see  below),  except  that  they  occur  after  an  error  and  refer  to 
the  goal  the  student  should  ^ave  been  following. 

Well,  you’ll  want  to  use  and  and  or,  not  two  or  s. 

Oh  wait,  you  don’t  want  to,  you  don't  want  to  return  now. 


Tutor  Surface  Feature  Feedback :  This  is  another  type  of  tutorial  error  feedback.  This 
type  of  feedback  points  the  student  to  the  feature  of  the  solution  that  is  incorrect.  The 
feature  could  be  syntactic  or  it  could  be  relating  to  the  function  the  student  chose,  and  so 
forth.  The  identifying  elements  of  this  category  are  that  an  error  has  occurred,  and  that 
the  tutor  simply  makes  the  student  aware  of  the  surface  feature  that  is  wrong. 

Well,  actually,  listp  would  return  true  for  an  empty  list,  also. 

Quote  why  is  not  a  function. 

The  tutor  attempts  to  assess  the  student  s  understanding  of  a  topic 

Tutor  Probe:  The  tutor  tries  to  determine  what  the  student  knows  about  some  topic. 
The  topic  could  be  a  LISP  function,  a  problem,  and  so  forth. 

OK,  now  do  you  remember  that? 

OK.  so  how  many  elements? 
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Tutor  Prompt:  This  category  is  for  tutor  utterances  that  are  asking  for  the  student  s 
next  step.  So  the  tutor  could  be  asking  for  the  next  step  of  a  problem,  of  an  example,  or 
of  the  understanding  of  a  problem. 

So  what  will  that  part<return? 

Cons  that  atom  into  the  list,  and  then  [pause]? 

The  tutor  helps  the  student  check  the  current  answer 

Tutor  Simulate  Process:  This  is  the  tutorial  version  of  Student  Simulate  Process,  and 
includes  utterances  that  include  the  production  of  data  in  the  manner  that  LISP  would. 
That  is,  the  tutor  works  through  the  behavior  the  computer  would  execute  on  the  LISP 
code;  this  could  be  verbal  or  non-verbal.  If  the  tutor  produces  actual  data  output  from  a 
function,  the  code  must  have  been  run  in  the  tutor's  head,  so  the  utterance  is  categorized 
here. 

So  this  is  not  a  list,  and  it  returns  nil 

If  you  call  the  function  atom,  on  nil.  it  returns  true,  because  it  says  that  nil  is 

an  atom.  It  also  returns  nil  if  we  do  it  on  listp. 

Miscellaneous  non-task-related  utterances 

Tutor  Comment:  This  is  the  tutorial  version  of  Student  Comment.  This  category 
consists  of  tutor  utterances  which  were  either  unrelated  to  the  task  or  uninterpretable. 

Whoops,  let  me  turn  this  off. 

Do  you  want  a  drink? 
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Appendix  C:  Transitions  between  events 


Insert  Table  14  about  here 


< 
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Table  1:  Transcript  of  a  LISP  tutoring  session 

1.  Student:  [typingj 

(defun  classify-sentence  (sent) 

2.  Tutor:  So,  that’s  a  very  long  function  name! 

3.  Student:  [typing] 

(cond  ((or  (equal  (car  sent)  ’why) 

(equal  (car  sent)  ’how))  ’question) 

((and  (member  ’was  sent) 

(member  ’bv  sent))  ’passive) 

(t  ’active))) 

All-rightv.  Yeah,  [typing]  (classify  sentence  (marv 

4.  Tutor:  You  have  to  quote  the  thing  here,  or  else  it’ll  think  it  s  a  function 
call,  [pause]  They  give  you  some  examples,  if  you  wanna  use  theirs.= 

5.  Student:  =Oh,  sure.=  [typing] 

<DEL><DEL><DEL><DELxDEL>’(mary  threw  the  snowball  at  Steve)) 

6.  Tutor:  =  or  you  could  just  make  up  some. 

7.  [Computer  returns  “Undefined  variable  sentence’] 

8.  Student:  Oops,  (pause)  Oh,  I  forgot  to  put  the  dash. 

9.  Tutor:  Yup!  So  it  thought  it  was  a  variable. 

10.  Student:  (typing]  (classify-sentence  ’(mary  threw  the  snowball  at  steve)) 
Nice! 

11.  Tutor:  Good. 

12.  Student:  I  think  I’ll  try  one  more  now. 

13.  Tutor:  Do  you  understand  the  difference  between  and  and  or  now? 

14.  Student:  Uhh  ...  let’s  see. 

15.  Tutor:  [unintell]  // 

16.  Student:  If  ..  and  needs  both  of  them  to  be  true,  and  then  it  returns  true. 

17.  Tutor:  Um  hmm. 

18.  Student:  And  or  just  needs  one  of  them  to  be  true,  and  it  returns  true. 

19.  Tutor:  Right. 

21.  Student:  But  if  both  of  them  are  nil  in  or,  then  it  would  return  nil. 

22.  Tutor:  Right.  And  in  both  of  them,  or  and  and ,  it  doesn’t  necessarily  return 
the  letter  t.  It’ll  return  whatever  true  value  that  it  gets  to. 

23.  Student:  Uhh  ...  I  wonder  -  I  wonder  how  that  worked  in  my  function  that 
I  just  wrote. 

24.  Tutor:  That’s  fine,  because  the  cond  knows,  cond  knows  that  anything 
that’s  not  nil  is  like  true. 
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Table  2: 
1.  TFA 


2.  SC 

3.  Comm 

4.  SC 

5.  TSP 


6.  SC 

7.  TSG 

8.  SC 

9.  TElab 

10.  TFA 

11.  TSP 


12.  SC 

13.  TSP 


14.  TSP 


15.  SC 


A  student-tutor  interaction  after  the  segmentation  and  categorization  process 

Tutor:  So  in  this  example  down  here?  See  how  they  have  two 
separate^ 

Student:  Yeah. 

Tutor:  =paramet£rs. 

Student:  Yeah. 

Tutor:  . . .  So.  if  you 
Student:  That  makes  sense. 

Tutor:  called  insert- second  on,  like,  . . .  dog ,  and  then  the  list  (bird 
cat  egg), 

Student:  Hmm. 

Tutor:  =then  item  would  always  refer  to  dog ,  for  your  function 
call,= 

Student:  Right. 

Tutor:  =and  oldlist  would  always  refer  to  that  list,  bird,  cat, 
whatever. 

Student:  Okay. 

Tutor:  And  then  you  have  to  figure  out  what  exactly  you  want  it 
to  do. 

Student:  Right. 

Tutor:  Using  those  functions  we  learned  yesterday,  [pausej 
Tutor:  This  is  a,  this  is  a  good  example,  I  like  this  . . .  thing.  'Cause 
this  shows  how  LISP  is  actually  going  through  and  interpreting  it. 

Tutor:  So,  let's  say  you  typed  in  this,  umm,  function  call-  function 
definition  of  double.  Telling  you  the  parameter  is  num ,  so  there  s 
only  one  parameter,  you're  only  going  to  have  one  argument.  But 
then  if  they  call,  if  you  call  double ,  on  this  other  function,  it  s  kind 
of  interesting  to  see  how  it  actually  evaluates  that  because  this 
whole  list,  {+  5  10),  is  going  to  eventually  be  assigned  to  num. 

Student:  Okay. 

Tutor:  Umm.  but  first,  okay,  it  looks  at  double ,  it  knows  this  is  the 
definition  it's  gonna  use.  and  then  it  has  to  evaluate  that  argument. 

So  it  works  inside-out,  like  it  did,  like  we  were  looking  at  yesterday. 

Tutor:  It  figures  out  what  five  plus  ten  is,  gets  15,  then  it  assigns 
15  to  num,  binds  num 
Student:  Mrara  hmm. 

Tutor:  to  15,  that’s  the  words  they  use.  and  uhh,  ...so  then,  in 
the  rest  of  the  body,  [laughsj  that  one  line,  num  substitutes-is 
substituted  with  15.  So  it  looks  at  the  body,  (*  num  2),  num  is 
evaluated  and  num  gets  15.  2  stays  itself,  and  then  it  applies  the 
multiplication,  multiplies  15  by  2,  and  this  line  returns  30.  Now 
whatever  the  body  of  the  function  returns,  the  whole  function  will 
return,  so  actually  30  will  get  printed  out  there. 

Student:  That  makes  sense. 
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Table  3:  Categories  of  student  actions  in  the  student-tutor  discourse 

•  The  student  constructs  a  solution  to  a  problem  (Student  Problem  Solving  Action). 

-  Student  Correction  [SCrj 

-  Student  Elaboration  [SElabi 

-  Student  Example  ISE] 

-  Student  Focus  Attention  [SFAl 

-  Student  Indicate  Difficulty  [SID] 

—  Student  Indicate  Lack  of  Understanding  [ILU] 

-  Student  Read  [Read] 

-  Student  Refer  [SRefer] 

-  Student  Set  Goal  [SSG] 

-  Student  Type  [Type] 

•  The  student  asks  for  help  from  the  tutor. 

-  Assist  Plan  Assertion  [APA] 

-  Assist  Plan  Question  [APQ] 

-  Assist  Understanding  [AU1 

-  Student  Informational  Request  [SIR] 

•  The  student  indicates  that  the  tutor's  utterances  were  understood. 

-  Student  Confirmation  [SC] 

•  The  student  checks  the  current  answer. 

-  Student  Simulate  Process  [SSP] 

•  Miscellaneous  non-task-related  utterances. 

-  Student  Comment  [Comm] 
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Table  4:  Categories  of  tutorial  actions  in  the  student-tutor  discourse 
< 

•  The  tutor  performs  a  portion  of  the  problem  solving. 

-  Tutor  Example  [TE] 

-  Tutor  Focus  Attention  [TFA] 

-  Tutor  Read  (Read) 

-  Tutor  Refer  [TReferj 

-  Tutor  Type  [Type] 

•  The  tutor  offers  guidance  for  the  student's  ongoing  problem  solving. 

-  Tutor  Confidence  Builder  [CB] 

-  Tutor  Hint  [Hint] 

-  Tutor  Indicate  Difficulty  [TID] 

-  Tutor  Set  Goal  [TSG] 

-  Tutor  Supportive  Statement  [SS] 

•  The  tutor  confirms  a  student  step  (TCS). 

-  Tutor  Confirmation  [TC] 

-  Tutor  Elaboration  [TElab] 

•  The  tutor  gives  error  feedback  after  an  incorrect  student  step. 

-  Tutor  Correction  [TCr] 

-  Tutor  Plan  Based  Feedback  [PBFi 

-  Tutor  Surface  Feature  Feedback  [SFF] 

•  The  tutor  attempts  to  assess  the  student's  understanding  of  a  topic. 

-  Tutor  Probe  [Probe] 

-  Tutor  Prompt  [Prompt] 

•  The  tutor  helps  the  student  check  the  current  answer. 

-  Tutor  Simulate  Process  [TSP] 

•  Miscellaneous  non-task-related  utterances. 

-  Tutor  Comment  [Comm] 
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Table  5:  A  listing  of  each  error  category  used,  its  frequency  of  occurrence,  and  its  definition 


Error  type 

Frequency 

Category  definition 

Typographical 

360 

A  typing  error,  involving  a  misspelling  or  an  illegal 
keystroke. 

Syntactic 

435 

The  addition  of  an  unneeded  parenthesis  or  quo 
tation  mark  or  the  deletion  of  one  that  is  needed. 

Semantic:  Operator 

123 

Asserting  that  a  function  does  something  it  does 
not  do  or  attempting  to  apply  a  function  when  it 
can  not  be  applied. 

Semantic:  Concept 

59 

An  error  relating  to  the  concepts  atom,  list,  nil 
variable,  or  elements  (of  a  list). 

Goal:  Incorrect 

178 

Stating  a  goal  to  achieve  that  is  not  needed  in 
the  problem  or  will  not  help  the  student  solve  the 
problem. 

Goal:  Skipped  Goal 

69 

Skipping  a  goal  that  is  needed  in  the  problem. 
This  could  occur  when  setting  up  an  initial  goal 
structure  or  when  solving  the  problem,  and  re¬ 
quires  explicit  evidence  that  the  student  has  failed 
to  achieve  some  subgoal. 
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Table  6:  Examples  of  Tutor  Confirmations  in  ongoing  problem  solving 


1.  SPSA 

2.  SPSA 

3.  TCS 

4.  TID 

5.  SPSA 

6.  TCS 

7.  SPSA 

8.  TCS 

9.  SPSA 

10.  SPSA 

11.  TCS 


Student:  So,  you  do  defun.  um.  first-elem. 
Student:  [typing]  (de fun  first-elem 
Tutor:  Right,  that’s  the  function  name. 
Tutor:  Now  comes  the  tough  part. 
Student:  Now  comes  the  parameters. 
Tutor:  The  parameter  list,  right. 

Student:  So,  it  just  needs  to  have  a  list. 
Tutor:  Um  hum. 

Student:  It  would  just  be  list. 

Student:  [types]  (list) 

Tutor:  Sure. 


91 


Tutoring:  Guided  learning  by  doing 


Table  7:  A  typical  example  of  an  error  flagged  by  the  student. 

1.  SPSA  Student:  [typing]  palp  (a  b  c  c  b  a)) 

[error:  there  needs  to  be'  a  quotation  mark  before  the  list) 

2.  SCr  Student:  Oh,  I  didn’t  put  the,  uh  // 

[flag  for  previous  step] 

3.  TElab  Tutor:  Quote. 

4.  SC  Student:  Quotes. 

5  TCS  Tutor:  Right. 

6.  SPSA  Student:  (types  repairj 
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Table  8:  An  example  of  the  tutor  flagging  a  student  error 

1.  SPSA  Student:  [Pause.]  So  you  could  use  or.  [unintel.]  Cond.  [Pause.] 

Um,  equal  [pause]  argl,  true  [pause] 

[error:  the  argument  may  not  be  equal  to  true] 

2.  SFF  Tutor:  Right.  But  what  if  it  was,  like,  the  atom  dog .  That  counts  as  true= 

[flag  for  error  in  step  1] 

3.  SC  Student:  =Uh-huh. 
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Table  9:  An  example  of  a  Tutor  Correction  following  a  Syntactic  Error 

1.  SPSA  Student:  [typing]  (back  (a  b  c) 

[error:  should  have  been  ’(a  b  c)\ 

2.  TCr  Tutor:  Now  remember  to  quote  that= 

Student:  =Umm.= 

=argument.  (a  b  c). 

3.  SPSA  Student:  [Begins  typing  repair] 

<DEL><DEL>. . .  <DEL>  ’(a  b  c)) 
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Table  10:  An  example  of  a  Tutor  Surface  Feature  Feedback  following  a  Semantic  Error 


1.  SPSA 

2.  SPSA 

3.  TSG 

4.  SPSA 

5.  SPSA 

6.  SFF 

7.  SPSA 

8.  TCS 

9.  Comm 

10.  Prompt 

11.  SPSA 

12.  TCS 

13.  SPSA 

14.  SPSA 


Student:  [types]  (defun  numhne  (num) 

Student:  So  you’re  putting  together  two  lists  if  [laugh] 

Tutor:  If  the  first  one  is  zero. 

Student:  If  number  is  zero.  And  nil  otherwise  . . . 

Student:  [types]  (append 

[error:  append  can  not  be  applied  to  non-list  arguments] 
Tutor:  Remember  append,  what  ap  . . . 

What  append’s  arguments  must  be,  though. 

Student:  Oh,  two  lists. 

Tutor:  Right 
Student:  So  [laughs] 

Tutor:  ’Cause  if,  if  it  comes  out  with  t= 

Student:  =t.  it’s  not  going  to  be  a  list 
Tutor:  It’s  not  a  list,  right. 

Student:  OK.  then  I  just  have  to  make,  have  to  create  a  list. 
Student:  [types]  <DEL><DEL>. . .  <DEL>ks£ 

[Error  in  step  5  is  repaired  here.] 
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Table  11:  An  example  of  a  Tutor  Plan  Based  Feedback  following  a  Goal  Error 


1.  SPSA 


2.  PBF 

3.  SPSA 

4.  TCS 

5.  TCS 

6.  SC 

7.  TElab 


Student:  [typing] 

(defun  carlis  ( oldlist ) 

(cond  ((listp  oldlist)  (car  oldlist 
[error:  this  is  not  the  first  case] 

Tutor:  eh  um,  a  good,  uh,  kinda  like  a  good  thing  to  keep  in  mind  then  in 
ordering  is  to  put  the,  most  specific  thing  first  . . .  most  specific  test  first. 
Student:  [Begins  typing  repair,  deletes  first  case] 

((null  oldlist)  'nil) 

Tutor:  Right,  this  is  a  case  where  you  do  need  two  parens. 

Um,  it’s  kind  unique,  because  the  first  paren  is  the  starting  of  the  case,  and 
then  the  second  one  is  for  the  function  that  you  are  about  to  call. 

Student:  ok  ok 

Tutor:  If  you  call  a  function. 
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Table  12: 
1. 

a. 

b. 

c. 

2. 

a. 

b. 

c. 

3. 

a. 

b. 

c. 


< 


Examples  of  all  three  types  of  explicit  error  feedback  for  different  error  types 

Syntactic  error:  Student  types  (member  item  Us)  -  there  should  not  be  a 
quotation  mark  before  the  Us. 

Tutor  Correction:  “You  should  remove  the  quote  before  Us  ’ 

Tutor  Surface  Feature  Feedback:  “Remember,  a  quoted  atom  is  treated 
literally,  it’s  not  evaluated.” 

Tutor  Plan-Based  Feedback:  UI  think  you  wanted  to  look  for  item  in  the 
value  of  Zis,  not  in  the  atom  Us. 

Semantic  Error:  When  attempting  to  get  the  element  following  some  tar¬ 
get  item  in  a  list,  the  student  types  (car  (member  item  Us )),  apparently 
forgetting  that  member  returns  a  list  of  item  and  all  items  following  it  in  Us. 

Tutor  Correction:  “You  want  the  car  of  the  cdr  of  the  return. 

Tutor  Surface  Feature  Feedback:  “Remember  member  returns  the  tail  of  the 
list  starting  with  the  item.” 

Tutor  Plan-Based  Feedback:  “Didn  t  you  want  to  get  the  element  after 
itemV 

Goal  Error:  When  trying  to  see  if  two  items  are  both  lists  in  the  context  of 
a  larger  problem,  the  student  types  ( or  (or  (listp  argl)  (listp  arg2))),  but 
the  inside  or  should  be  an  and. 

Tutor  Correction:  “The  second  or  should  be  an  and” 

Tutor  Surface  Feature  Feedback:  “You  don’t  want  two  or’s,  do  you? 

Tutor  Plan-Based  Feedback:  “You  meant  to  see  if  both  things  were  lists,  not 
if  either  one  was  a  list.” 
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1.  SPSA 


2.  TC 
3  SPSA 


4.  PBF 


5.  SPSA 

6.  TCS 

7.  TID 

8.  SPSA 


< 


Table  13:  An  example  of  a  collaborative  repair  of  a  student  error 

Student:  [typing] 

(defun  classify  (var) 

(cond  ((null  var)  ’nil) 

((numberp  var)  ’number) 

Tutor:  Right,  um  hum,  just  the  word  number. 

Student:  [continues  typing] 

((t 

[error:  there  should  be  only  one  parenthesis.] 

Now  here  you  have  the  choice.  You  could  either  use  t= 

Student:  =t 
or  the  predicate= 

Student:  ok 
or  list. 

Student:  So  I  don’t  need  that  extra. 

Tutor:  Right,  you  got  it,  you  got  it. 

Tutor:  That’s  kinda  tough  cause  it’s  a  weird  pattern  that  these  conds  have. 
Great. 

Student:  [typing] 

<DEL><DEL>f 
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Table  14:  The  numbers  in  this  table  are  the  frequency  of  transitions  between  events.  The 
labels  along  the  left  side  are  the  first  event,  and  the  labels  along  the  top  are  the  subsequent 
event.  For  example,  the  leftmost  number  of  the  second  row  is  34.  This  number  indicates 
that  there  were  34  instances  where  a  Student  Problem  Solving  Action  followed  a  Student 
Ask  for  Help. 


S  Ask 

S 

S  Check 

Tutor 

Tutor 

Tutor 

T  Error 

T  Assess 

T  Check 

SPSA 

or  Help 

Confirm 

Answer 

PS  Action 

Guidance 

Confirm  Step 

FB 

Und 

Answer 

SPSA 

623 

136 

30 

75 

153 

513 

1744 

397 

156 

78 

S  Ask  for  Help 

34 

0 

6 

3 

19 

68 

501 

67 

23 

23 

S  Confirm 

2S2 

56 

33 

29 

130 

311 

807 

36 

92 

201 

S  Check  Aniwer 

40 

1 

4 

28 

10 

25 

249 

59 

17 

33 

Tutor  PS  Action 

120 

2 

168 

21 

15 

18 

42 

17 

15 

T  Guidance 

561 

63 

343 

30 

29 

93 

112 

30 

33 

30 

Tutor  Confirm  Step 

1495 

302 

923 

168 

72 

273 

493 

47 

58 

118 

T  Error  FB 

281 

43 

222 

25 

14 

48 

64 

14 

18 

17 

T  Assess  Und 

37 

26 

88 

80 

4 

15 

22 

5 

6 

u 

T  Check  Answer 

23 

124 

230 

33 

14 

7 

22 

61 

9 

2-i 

TOTAL 

3506 

753 

2047 

495 

460 

1371 

4056 

733 

427 

552 

99 


Tutoring:  Guided  learning  by  doing 


Figure  Captions 

Figure  1.  A  presentation  of  student  and  tutor  actions  and  their  chronological  rela¬ 
tionships.  The  half  circles  represent  tutor  events,  and  the  squares  within  circles  represent 
student  events.  The  arrows  Connecting  objects  describe  which  events  followed  other  events 
in  the  data.  Darker  links  indicate  high  frequency  transitions  that  occurred  more  than  100 
times. 

Figure  2.  The  number  of  events  that  intervened  between  a  student  error  and  the  flag¬ 
ging  of  that  error  by  the  student  or  the  tutor.  A  lag  of  zero  indicates  that  the  error  was 
flagged  during  the  same  event  within  which  it  occurred. 

Figure  3.  The  frequency  of  tutorial  feedback  that  occurred  in  response  to  each  type  of 
student  error. 

Figure  4.  The  median  number  of  events  required  to  repair  each  of  the  different  types 
of  errors.  Longer  repairs  indicate  more  difficult  or  more  collaborative  repairs. 

Figure  5.  The  sequence  of  student  and  tutor  collaborative  actions  when  repairing  an 
error.  The  dotted  lines  represent  infrequent  transitions,  less  than  100  times  out  of  the  over 
1,000  errors  represented. 
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